[jira] [Commented] (CASSANDRA-15529) AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16 to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031245#comment-17031245
]
feroz shaik commented on CASSANDRA-15529:
-
[~ifesdjeen] - Let us know if you need any further information. Thanks for the
support.
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
> to 3.11.4
> -
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: Pooja Nair
>Priority: Urgent
> Labels: 2.1.16, 3.11.4
> Attachments: sstable_dump.txt
>
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND
> durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as
> [shardid][salid][timestamp]. The table records SALIDs to be deleted along
> with their deletion times (which may be modified)'
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> Things to note:
> # The column2 is always passed a null value during insertion
> # column2 is a part of primary key
> # Range select and Range delete is done through our app.
> Iniatally the cluster was on casssandra version 2.1.16 and have been
> recently upgraded to 3.11.4 post the upgrade, we see that the nodes are going
> down, and log the below exceptions during startup and even after node is up.
> This one node is causing the whole cluster to behave improperly.
> {code:java}
> WARN [Native-Transport-Requests-47] 2020-01-29 13:49:05,190
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[Native-Transport-Requests-47,5,main]: {} java.lang.RuntimeException:
> java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has
> an open RT bound as its last item at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2588)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0-internal] at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.concurrent.SEPExecutor.maybeExecuteImmediately(SEPExecutor.java:194)
> [apache-cassandra-3.11.4.jar:3.11.4] at
>
[jira] [Comment Edited] (CASSANDRA-15529) AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16 to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029481#comment-17029481
]
feroz shaik edited comment on CASSANDRA-15529 at 2/4/20 2:35 AM:
-
Hi [~ifesdjeen] , I am a colleague of [~Pooja_nair] and would like to answer
your questions. # We have all the nodes upgraded to 3.11.4 and also finished
upgradesstables. There are no previous sstables present on any of the
nodes/cluster.
# We have no clue whether the affected node is cordinator or not as it is too
random and the affected node behaves rogue potentially impacting whole cluster.
# The query patterns are like below:
static String INSERT_SCHEDULED_DELETION_QUERY = "INSERT INTO \"sal_purge\"
(key,column1,value) VALUES (?,?,?) USING TIMESTAMP
?;"; static String SELECT_SCHEDULED_DELETION_QUERY = "SELECT column1, value
FROM sal_purge where key=? AND column1>=? LIMIT
?;"; static String DELETE_SCHEDULED_DELETION_QUERY = "DELETE FROM
\"sal_purge\" USING TIMESTAMP ? WHERE key=? AND column1=?;";
I will try to upload the sstable dump soon to this, Appreciate your support
[~ifesdjeen]
was (Author: ferozshaik...@gmail.com):
Hi [~ifesdjeen] , I am a colleague of [~Pooja_nair] and would like to answer
your questions. # We have all the nodes upgraded to 3.11.4 and also finished
upgradesstables. There are no previous sstables present on any of the
nodes/cluster.
# We have no clue whether the affected node is cordinator or not as it is too
random and the affected node behaves rogue potentially bring impacting whole
cluster.
# The query patterns are like below:
static String INSERT_SCHEDULED_DELETION_QUERY = "INSERT INTO \"sal_purge\"
(key,column1,value) VALUES (?,?,?) USING TIMESTAMP
?;"; static String SELECT_SCHEDULED_DELETION_QUERY = "SELECT column1, value
FROM sal_purge where key=? AND column1>=? LIMIT
?;"; static String DELETE_SCHEDULED_DELETION_QUERY = "DELETE FROM \"sal_purge\"
USING TIMESTAMP ? WHERE key=? AND column1=?;";
I will try to upload the sstable dump soon to this, Appreciate your support
[~ifesdjeen]
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
> to 3.11.4
> -
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: Pooja Nair
>Priority: Urgent
> Labels: 2.1.16, 3.11.4
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND
> durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as
> [shardid][salid][timestamp]. The table records SALIDs to be deleted along
> with their deletion times (which may be modified)'
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
>
[jira] [Commented] (CASSANDRA-15529) AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16 to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029481#comment-17029481
]
feroz shaik commented on CASSANDRA-15529:
-
Hi [~ifesdjeen] , I am a colleague of [~Pooja_nair] and would like to answer
your questions. # We have all the nodes upgraded to 3.11.4 and also finished
upgradesstables. There are no previous sstables present on any of the
nodes/cluster.
# We have no clue whether the affected node is cordinator or not as it is too
random and the affected node behaves rogue potentially bring impacting whole
cluster.
# The query patterns are like below:
static String INSERT_SCHEDULED_DELETION_QUERY = "INSERT INTO \"sal_purge\"
(key,column1,value) VALUES (?,?,?) USING TIMESTAMP
?;"; static String SELECT_SCHEDULED_DELETION_QUERY = "SELECT column1, value
FROM sal_purge where key=? AND column1>=? LIMIT
?;"; static String DELETE_SCHEDULED_DELETION_QUERY = "DELETE FROM \"sal_purge\"
USING TIMESTAMP ? WHERE key=? AND column1=?;";
I will try to upload the sstable dump soon to this, Appreciate your support
[~ifesdjeen]
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
> to 3.11.4
> -
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: Pooja Nair
>Priority: Urgent
> Labels: 2.1.16, 3.11.4
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND
> durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as
> [shardid][salid][timestamp]. The table records SALIDs to be deleted along
> with their deletion times (which may be modified)'
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> Things to note:
> # The column2 is always passed a null value during insertion
> # column2 is a part of primary key
> # Range select and Range delete is done through our app.
> Iniatally the cluster was on casssandra version 2.1.16 and have been
> recently upgraded to 3.11.4 post the upgrade, we see that the nodes are going
> down, and log the below exceptions during startup and even after node is up.
> This one node is causing the whole cluster to behave improperly.
> {code:java}
> WARN [Native-Transport-Requests-47] 2020-01-29 13:49:05,190
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[Native-Transport-Requests-47,5,main]: {} java.lang.RuntimeException:
> java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has
> an open RT bound as its last item at
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939183#comment-16939183
]
feroz shaik commented on CASSANDRA-15263:
-
I figured out that the app deletes using : "DELETE FROM \"sal_purge\" USING
TIMESTAMP ? WHERE key=? AND column1=?;"; is actually causing range bound
tombstones. I used sstabledump to know that.
"rows" : [
{
"type" : "range_tombstone_bound",
"start" : {
"type" : "inclusive",
"clustering" : [
"15e0088cdb99f399290e531a452e4c2c32d547a852607cb2819ff8fbd565ed53", "*" ],
"deletion_info" : \{ "marked_deleted" : "2019-09-27T06:20:16.04Z",
"local_delete_time" : "2019-09-27T06:38:02Z" }
}
},
But other problems i have is that the sstabledump is erroring out with below
which may be another problem.bug itself:
Exception in thread "main" java.lang.NullPointerException
at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:156)
at org.apache.cassandra.db.marshal.UTF8Type.toJSONString(UTF8Type.java:66)
at
org.apache.cassandra.tools.JsonTransformer.serializeClustering(JsonTransformer.java:347)
at
org.apache.cassandra.tools.JsonTransformer.serializeRow(JsonTransformer.java:244)
at
org.apache.cassandra.tools.JsonTransformer.serializePartition(JsonTransformer.java:211)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at org.apache.cassandra.tools.JsonTransformer.toJson(JsonTransformer.java:101)
at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:240)
My main point of concern now is that I cannot reproduce the error on my lab by
cqlsh driver with CL-all.
[~benedict] - since we know this is for sure a bug , when can we expect any
fix/patch for this? Kindly let us know.
> LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
> ---
>
> Key: CASSANDRA-15263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15263
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: feroz shaik
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Labels: 2.1.16, 3.11.4
> Attachments: sample.system.log, schema.txt,
> sstabledump_sal_purge_d03.json, sstablemetadata_sal_purge_d03,
> stack_trace.txt, system.log, system.log, system.log, system.log,
> system_latest.log
>
>
> We have hit a problem today while upgrading from 2.1.16 to 3.11.4.
> we encountered this as soon as the first node started up with 3.11.4
> The full error stack is attached - [^stack_trace.txt]
>
> The below errors continued in the log file as long as the process was up.
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
>
> The nodetool version says 3.11.4 and the no of connections on native por t-
> 9042 was similar to other nodes. The exceptions were scary that we had to
> call off the change. Any help and insights to this problem from the community
> is appreciated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938558#comment-16938558
]
feroz shaik commented on CASSANDRA-15263:
-
Dear [~benedict] & Shalom,
I have been working on this lately with my engineering and business to move
with upgrade task but there is some ambiguity and checking with you if you can
assist. We want to be absolutely sure that post upgrade the app continues to
work as normal and this can be only answered by providing test results. So i
have started testing this on my lab (3 nodes).
I was able to reproduce the error with some other table but cant replicate the
problem using my table that app uses.
I got the exact query patterns from the application team by enabling debug:
static String INSERT_SCHEDULED_DELETION_QUERY = "INSERT INTO \"sal_purge\"
(key,column1,value) VALUES (?,?,?) USING TIMESTAMP ?;"; static String
SELECT_SCHEDULED_DELETION_QUERY = "SELECT column1, value FROM sal_purge where
key=? AND column1>=? LIMIT ?;"; static String DELETE_SCHEDULED_DELETION_QUERY =
"DELETE FROM \"sal_purge\" USING TIMESTAMP ? WHERE key=? AND column1=?;";
My table structure if you remember:
CREATE TABLE "SAL".sal_purge (CREATE TABLE "SAL".sal_purge ( key text,
column1 text, column2 text, value text, PRIMARY KEY (key, column1,
column2) ) WITH COMPACT STORAGE AND CLUSTERING ORDER BY (column1 ASC,
column2 ASC) AND bloom_filter_fp_chance = 0.1 AND caching =
'\{"keys":"NONE", "rows_per_partition":"NONE"}' AND comment = 'Holds items
to be removed as [shardid][salid][timestamp]. The table records SALIDs to be
deleted along with their deletion times (which may be modified)' AND
compaction = \{'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} AND
compression = \{'chunk_length_kb': '64', 'sstable_compression':
'org.apache.cassandra.io.compress.SnappyCompressor'} AND
dclocal_read_repair_chance = 0.0 AND default_time_to_live = 0 AND
gc_grace_seconds = 864000 AND max_index_interval = 2048 AND
memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND
read_repair_chance = 0.1 AND speculative_retry = '99.0PERCENTILE';
**I really cannot understand how can this "DELETE FROM \"sal_purge\" USING
TIMESTAMP ? WHERE key=? AND column1=?;"; cause a range tombstone..I can only
see they can be row deletions isnt it?
Now, I tried to insert some data, do some delete, upgrade 1st node while i hit
a range select from node 2. I am still unable to replicate the problem.
cqlsh> select * from "SAL".sal_purge;cqlsh> select * from "SAL".sal_purge; key
| column1 | column2 |
value
-+--+-+---
15e | 15e000b946229403b2010e542724cb9af7b939df0c5dd7c5cb680b28881b0905 |
null | 1568628121530 15e |
15e000cdbcadb3ea81fe66a2023ccae8dde2cd1c0cdcd25c1b011f5cf4520411 | null |
1568661805145 15e |
15e000d5d1b5ea0e692046b51901d6efe7e1e9beb1f873f5bf62cd9b8b78b6a7 | null |
1568717250061 15e |
15e00205a8a8beafb571f0824022061975f239ffdfec02ae5ec2282d6db76e5b | null |
1568776769724 15e |
15e00322575230ed208cfa101afac66e790582ecd179339e16ee811d41bbac08 | null |
1568755358394 15e |
15e003fb651375db41cbcba3c37f0ab02f9308ba2e3708a5e7be359189583f26 | null |
1568630537539 15e |
15e0049bce42be7e46c7beb6f8f878abd83350556de3864477e94fc33643e818 | null |
1568675019827 15e |
15e005cb926ca2b723260927c3ba9d16ee8092d6adb78d01e129815617e26251 | null |
1568642160143 15e |
15e007f24dd7c31358a23869b06fbc24095e133cdee9cc9e5af88e2a836e9c03 | null |
1568749318982 15e |
15e0088cdb99f399290e531a452e4c2c32d547a852607cb2819ff8fbd565ed53 | null |
1568690314042 (10 rows)
cqlsh> DELETE FROM "SAL".sal_purge USING TIMESTAMP 400 where key='15e' and
column1='15e0049bce42be7e46c7beb6f8f878abd83350556de3864477e94fc33643e818';cqlsh>
DELETE FROM "SAL".sal_purge USING TIMESTAMP 400 where key='15e' and
column1='15e0049bce42be7e46c7beb6f8f878abd83350556de3864477e94fc33643e818';
cqlsh> DELETE FROM "SAL".sal_purge USING TIMESTAMP 300 where key='15e' and
column1='15e005cb926ca2b723260927c3ba9d16ee8092d6adb78d01e129815617e26251';
cqlsh> DELETE FROM "SAL".sal_purge USING TIMESTAMP 200 where key='15e' and
column1='15e007f24dd7c31358a23869b06fbc24095e133cdee9cc9e5af88e2a836e9c03';
cqlsh> cqlsh> cqlsh> select key,column1,writetime(value) from "SAL".sal_purge
where key='15e'; key | column1
| writetime(value)
-+--+--
15e | 15e000b946229403b2010e542724cb9af7b939df0c5dd7c5cb680b28881b0905 |
1000 15e |
15e000cdbcadb3ea81fe66a2023ccae8dde2cd1c0cdcd25c1b011f5cf4520411 |
[jira] [Commented] (CASSANDRA-15274) Multiple Corrupt datafiles across entire environment
[
https://issues.apache.org/jira/browse/CASSANDRA-15274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911135#comment-16911135
]
feroz shaik commented on CASSANDRA-15274:
-
[~benedict]- can you please share that SSTableExport.Java (with CRC disabled)
which is 2.2 compatible if you have it handy. I was trying to add that line as
you suggested but had some issues.
> Multiple Corrupt datafiles across entire environment
> -
>
> Key: CASSANDRA-15274
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15274
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: Phil O Conduin
>Priority: Normal
>
> Cassandra Version: 2.2.13
> PRE-PROD environment.
> * 2 datacenters.
> * 9 physical servers in each datacenter - (_Cisco UCS C220 M4 SFF_)
> * 4 Cassandra instances on each server (cass_a, cass_b, cass_c, cass_d)
> * 72 Cassandra instances across the 2 data centres, 36 in site A, 36 in site
> B.
> We also have 2 Reaper Nodes we use for repair. One reaper node in each
> datacenter each running with its own Cassandra back end in a cluster together.
> OS Details [Red Hat Linux]
> cass_a@x 0 10:53:01 ~ $ uname -a
> Linux x 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64
> x86_64 x86_64 GNU/Linux
> cass_a@x 0 10:57:31 ~ $ cat /etc/*release
> NAME="Red Hat Enterprise Linux Server"
> VERSION="7.6 (Maipo)"
> ID="rhel"
> Storage Layout
> cass_a@xx 0 10:46:28 ~ $ df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/vg01-lv_root 20G 2.2G 18G 11% /
> devtmpfs 63G 0 63G 0% /dev
> tmpfs 63G 0 63G 0% /dev/shm
> tmpfs 63G 4.1G 59G 7% /run
> tmpfs 63G 0 63G 0% /sys/fs/cgroup
> >> 4 cassandra instances
> /dev/sdd 1.5T 802G 688G 54% /data/ssd4
> /dev/sda 1.5T 798G 692G 54% /data/ssd1
> /dev/sdb 1.5T 681G 810G 46% /data/ssd2
> /dev/sdc 1.5T 558G 932G 38% /data/ssd3
> Cassandra load is about 200GB and the rest of the space is snapshots
> CPU
> cass_a@x 127 10:58:47 ~ $ lscpu | grep -E '^Thread|^Core|^Socket|^CPU\('
> CPU(s): 64
> Thread(s) per core: 2
> Core(s) per socket: 16
> Socket(s): 2
> *Description of problem:*
> During repair of the cluster, we are seeing multiple corruptions in the log
> files on a lot of instances. There seems to be no pattern to the corruption.
> It seems that the repair job is finding all the corrupted files for us. The
> repair will hang on the node where the corrupted file is found. To fix this
> we remove/rename the datafile and bounce the Cassandra instance. Our
> hardware/OS team have stated there is no problem on their side. I do not
> believe it the repair causing the corruption.
>
> So let me give you an example of a corrupted file and maybe someone might be
> able to work through it with me?
> When this corrupted file was reported in the log it looks like it was the
> repair that found it.
> $ journalctl -u cassmeta-cass_b.service --since "2019-08-07 22:25:00" --until
> "2019-08-07 22:45:00"
> Aug 07 22:30:33 cassandra[34611]: INFO 21:30:33 Writing
> Memtable-compactions_in_progress@830377457(0.008KiB serialized bytes, 1 ops,
> 0%/0% of on/off-heap limit)
> Aug 07 22:30:33 cassandra[34611]: ERROR 21:30:33 Failed creating a merkle
> tree for [repair #9587a200-b95a-11e9-8920-9f72868b8375 on KeyspaceMetadata/x,
> (-1476350953672479093,-1474461
> Aug 07 22:30:33 cassandra[34611]: ERROR 21:30:33 Exception in thread
> Thread[ValidationExecutor:825,1,main]
> Aug 07 22:30:33 cassandra[34611]: org.apache.cassandra.io.FSReadError:
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /x/ssd2/data/KeyspaceMetadata/x-1e453cb0
> Aug 07 22:30:33 cassandra[34611]: at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:365)
> ~[apache-cassandra-2.2.13.jar:2.2.13]
> Aug 07 22:30:33 cassandra[34611]: at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361)
> ~[apache-cassandra-2.2.13.jar:2.2.13]
> Aug 07 22:30:33 cassandra[34611]: at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:340)
> ~[apache-cassandra-2.2.13.jar:2.2.13]
> Aug 07 22:30:33 cassandra[34611]: at
> org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:382)
> ~[apache-cassandra-2.2.13.jar:2.2.13]
> Aug 07 22:30:33 cassandra[34611]: at
> org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:366)
>
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911131#comment-16911131 ] feroz shaik commented on CASSANDRA-15263: - Cheers [~benedict] . we have now engaged our engineering to modify the table as to eliminate that dependency of null-column2. We are also looking at temporary options to shutdown the application that reads/writes to this table during the upgrade window. I will keep the chain posted with new developments from ours side. Thank you very much once again to help identify the issue. > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, > sstabledump_sal_purge_d03.json, sstablemetadata_sal_purge_d03, > stack_trace.txt, system.log, system.log, system.log, system.log, > system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15274) Multiple Corrupt datafiles across entire environment
[
https://issues.apache.org/jira/browse/CASSANDRA-15274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905808#comment-16905808
]
feroz shaik commented on CASSANDRA-15274:
-
Thank you [~philoconduin] . I just want to add other things to this problem
that we already went through for the community to be aware off.
# Power disruptions if any - Nothing of that sort reported by infra team.
# Storage related glitches/issues - Nothing.
# Network issues - Nothing. (we have not looked in detail with packet capture
and drops etc, but from monitoring it is clean).
# Schema change - It was reported on some forum that dropping a column and
re-creating it back with a different datatype could cause corruptions - This
was checked but there was no sort of such schema change on the cluster.
# CRC check - This is something we are still investigating. If CRC was not
being done effectively, there is another theory why it would only fail for
certain data files and not all? From what we have been seeing is that the
corruption could be on any CF, with no pattern to single compaction strategy
used etc..
Another important consideration to take into account is our PROD env which is
same like PRE-PROD in terms of infrastructure and C* config setup, schema. The
only difference is the amount of data residing there - its only 6-10G avg as
compared to 200 G avg'ng on pre-prod. We do not have any issues there (PROD).
> Multiple Corrupt datafiles across entire environment
> -
>
> Key: CASSANDRA-15274
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15274
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: Phil O Conduin
>Priority: Normal
>
> Cassandra Version: 2.2.13
> PRE-PROD environment.
> * 2 datacenters.
> * 9 physical servers in each datacenter - (_Cisco UCS C220 M4 SFF_)
> * 4 Cassandra instances on each server (cass_a, cass_b, cass_c, cass_d)
> * 72 Cassandra instances across the 2 data centres, 36 in site A, 36 in site
> B.
> We also have 2 Reaper Nodes we use for repair. One reaper node in each
> datacenter each running with its own Cassandra back end in a cluster together.
> OS Details [Red Hat Linux]
> cass_a@x 0 10:53:01 ~ $ uname -a
> Linux x 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64
> x86_64 x86_64 GNU/Linux
> cass_a@x 0 10:57:31 ~ $ cat /etc/*release
> NAME="Red Hat Enterprise Linux Server"
> VERSION="7.6 (Maipo)"
> ID="rhel"
> Storage Layout
> cass_a@xx 0 10:46:28 ~ $ df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/vg01-lv_root 20G 2.2G 18G 11% /
> devtmpfs 63G 0 63G 0% /dev
> tmpfs 63G 0 63G 0% /dev/shm
> tmpfs 63G 4.1G 59G 7% /run
> tmpfs 63G 0 63G 0% /sys/fs/cgroup
> >> 4 cassandra instances
> /dev/sdd 1.5T 802G 688G 54% /data/ssd4
> /dev/sda 1.5T 798G 692G 54% /data/ssd1
> /dev/sdb 1.5T 681G 810G 46% /data/ssd2
> /dev/sdc 1.5T 558G 932G 38% /data/ssd3
> Cassandra load is about 200GB and the rest of the space is snapshots
> CPU
> cass_a@x 127 10:58:47 ~ $ lscpu | grep -E '^Thread|^Core|^Socket|^CPU\('
> CPU(s): 64
> Thread(s) per core: 2
> Core(s) per socket: 16
> Socket(s): 2
> *Description of problem:*
> During repair of the cluster, we are seeing multiple corruptions in the log
> files on a lot of instances. There seems to be no pattern to the corruption.
> It seems that the repair job is finding all the corrupted files for us. The
> repair will hang on the node where the corrupted file is found. To fix this
> we remove/rename the datafile and bounce the Cassandra instance. Our
> hardware/OS team have stated there is no problem on their side. I do not
> believe it the repair causing the corruption.
>
> So let me give you an example of a corrupted file and maybe someone might be
> able to work through it with me?
> When this corrupted file was reported in the log it looks like it was the
> repair that found it.
> $ journalctl -u cassmeta-cass_b.service --since "2019-08-07 22:25:00" --until
> "2019-08-07 22:45:00"
> Aug 07 22:30:33 cassandra[34611]: INFO 21:30:33 Writing
> Memtable-compactions_in_progress@830377457(0.008KiB serialized bytes, 1 ops,
> 0%/0% of on/off-heap limit)
> Aug 07 22:30:33 cassandra[34611]: ERROR 21:30:33 Failed creating a merkle
> tree for [repair #9587a200-b95a-11e9-8920-9f72868b8375 on KeyspaceMetadata/x,
> (-1476350953672479093,-1474461
> Aug 07 22:30:33 cassandra[34611]: ERROR 21:30:33 Exception in thread
>
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905770#comment-16905770 ] feroz shaik commented on CASSANDRA-15263: - Thank you [~benedict] for those findings. I think what I will try to do next is to convince our customer that these messages logged are non-impacting exceptions and would not pose any failures for requests (read/write) . I hope my understanding is correct! In meanwhile, can we expect any interim fix for this issue from community? I once again convey my deep gratitude for all your support. > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, > sstabledump_sal_purge_d03.json, sstablemetadata_sal_purge_d03, > stack_trace.txt, system.log, system.log, system.log, system.log, > system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904858#comment-16904858 ] feroz shaik commented on CASSANDRA-15263: - sstablemetadata content of that file also attached - [^sstablemetadata_sal_purge_d03] > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, > sstabledump_sal_purge_d03.json, sstablemetadata_sal_purge_d03, > stack_trace.txt, system.log, system.log, system.log, system.log, > system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feroz shaik updated CASSANDRA-15263: Attachment: sstablemetadata_sal_purge_d03 > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, > sstabledump_sal_purge_d03.json, sstablemetadata_sal_purge_d03, > stack_trace.txt, system.log, system.log, system.log, system.log, > system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feroz shaik updated CASSANDRA-15263: Attachment: sstabledump_sal_purge_d03.json > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, > sstabledump_sal_purge_d03.json, stack_trace.txt, system.log, system.log, > system.log, system.log, system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904857#comment-16904857 ] feroz shaik commented on CASSANDRA-15263: - Thanks [~benedict]. I dont think V4 of that logger did log anything additional from previous. Attaching the latest system.log here - [^system.log]. sstable export is deprescated looks like in this version. I got you an equivalent sstabledump instead. Procedure followed: nodetool getendpoints "SAL" sal_purge d03 10.176.56.129 10.176.56.120 10.163.56.42 10.163.56.21 10.176.56.113 10.163.56.56 nodetool getsstables "SAL" sal_purge d03 /data3/cassandra/data/SAL/sal_purge-9c2a14b9fe873540a4fdc5183fc77c19/md-80142-big-Data.db sstabledump -d /data3/cassandra/data/SAL/sal_purge-9c2a14b9fe873540a4fdc5183fc77c19/md-80142-big-Data.db -k d03 >/var/log/sstabledump_sal_purge_d03.json The output of it is attached here - [^sstabledump_sal_purge_d03.json] > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt, > system.log, system.log, system.log, system.log, system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feroz shaik updated CASSANDRA-15263: Attachment: system.log > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt, > system.log, system.log, system.log, system.log, system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feroz shaik updated CASSANDRA-15263: Attachment: system.log > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt, > system.log, system.log, system.log, system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904623#comment-16904623 ] feroz shaik commented on CASSANDRA-15263: - Thank you [~benedict]. I thought the same aswell from looking at previous logs. I have now ran the latest version and captured the log here - [^system.log] > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt, > system.log, system.log, system.log, system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feroz shaik updated CASSANDRA-15263: Attachment: system.log > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt, > system.log, system.log, system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904405#comment-16904405 ] feroz shaik commented on CASSANDRA-15263: - Latest one attached. Thanks [^system.log] > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt, > system.log, system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903812#comment-16903812 ] feroz shaik commented on CASSANDRA-15263: - I added the latest system.log. Thanks once again [~benedict] Another step closer. We know which table is offending one now. > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt, > system.log, system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feroz shaik updated CASSANDRA-15263: Attachment: system_latest.log > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt, > system.log, system_latest.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903791#comment-16903791 ] feroz shaik commented on CASSANDRA-15263: - Sure . will update soon. > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt, > system.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feroz shaik updated CASSANDRA-15263: Attachment: system.log > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt, > system.log > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903736#comment-16903736 ] feroz shaik commented on CASSANDRA-15263: - [^system.log] This has additional log entries than previous one. > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903722#comment-16903722 ] feroz shaik commented on CASSANDRA-15263: - Thank you [~benedict]. I managed to build it locally and place that snapshot 3.11.4 version on the upgraded node. Below is the sample of the logs for your quick reference. I can identify the key from the output and trying to figure out which table it might belong to. I shall provide you soon the sstableexport to secure location. Please let me know where to do that. [^sample.system.log] > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feroz shaik updated CASSANDRA-15263: Attachment: sample.system.log > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: sample.system.log, schema.txt, stack_trace.txt > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903614#comment-16903614 ] feroz shaik commented on CASSANDRA-15263: - Hi Benedict, are we sure this build is 3.11.4 based? When i built it on the upgraded node and trying to start - I get this message "Cassandra 4.0 requires either Java 8 (update 151 or newer) or Java 11 (or newer). Java 8 update 131 is not supported." Can you kindly check please. > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: schema.txt, stack_trace.txt > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902955#comment-16902955
]
feroz shaik commented on CASSANDRA-15263:
-
I have a confirmation from the team that we had thrift schema before but we did
create new tables(cql) and migrated the data. So the problems related to thrift
and tombstone may be ruled out.
Also, as part of experimenting that single node that got upgraded to 3.11.4,
the upgradesstable step completed. The warnings still show up.
"WARN [ReadStage-4] 2019-08-08 09:04:55,157
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-4,5,main]: {}
java.lang.NullPointerException: null".
[~benedict] - It would be great if you can provide us that special build you
mentioned previously to get us the exact error. Thanks for the help.
> LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
> ---
>
> Key: CASSANDRA-15263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15263
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: feroz shaik
>Assignee: Benedict
>Priority: Normal
> Labels: 2.1.16, 3.11.4
> Attachments: schema.txt, stack_trace.txt
>
>
> We have hit a problem today while upgrading from 2.1.16 to 3.11.4.
> we encountered this as soon as the first node started up with 3.11.4
> The full error stack is attached - [^stack_trace.txt]
>
> The below errors continued in the log file as long as the process was up.
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
>
> The nodetool version says 3.11.4 and the no of connections on native por t-
> 9042 was similar to other nodes. The exceptions were scary that we had to
> call off the change. Any help and insights to this problem from the community
> is appreciated.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902005#comment-16902005 ] feroz shaik commented on CASSANDRA-15263: - Thank you Benedict. I am asking my developers about this info. Will revert soon. > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Assignee: Benedict >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: schema.txt, stack_trace.txt > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feroz shaik updated CASSANDRA-15263: Description: We have hit a problem today while upgrading from 2.1.16 to 3.11.4. we encountered this as soon as the first node started up with 3.11.4 The full error stack is attached - [^stack_trace.txt] The below errors continued in the log file as long as the process was up. ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 ErrorMessage.java:384 - Unexpected exception during request java.lang.NullPointerException: null ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 ErrorMessage.java:384 - Unexpected exception during request java.lang.NullPointerException: null ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 The nodetool version says 3.11.4 and the no of connections on native por t- 9042 was similar to other nodes. The exceptions were scary that we had to call off the change. Any help and insights to this problem from the community is appreciated. was: We have also hit a problem today while upgrading from 2.1.16 to 3.11.4. we encountered this as soon as the first node started up with 3.11.4 The full error stack is attached - [^stack_trace.txt] The below errors continued in the log file as long as the process was up. ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 ErrorMessage.java:384 - Unexpected exception during request java.lang.NullPointerException: null ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 ErrorMessage.java:384 - Unexpected exception during request java.lang.NullPointerException: null ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 The nodetool version says 3.11.4 and the no of connections on native por t- 9042 was similar to other nodes. The exceptions were scary that we had to call off the change. Any help and insights to this problem from the community is appreciated. > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: schema.txt, stack_trace.txt > > > We have hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901760#comment-16901760 ] feroz shaik edited comment on CASSANDRA-15263 at 8/7/19 6:23 AM: - Schema is attached - [^schema.txt] ^We current use CQL . Legacy mode was thrift.^ This cluster is one of the oldest and very first Cassandra implementations. so pretty legacy data in there.We also witness that it has tombstones for certain tables . We see them logged in the system.log. Could it be this specific behavior of converting this legacy data combined with tombstones thats causing the problem? Currently, we have removed that node from serving traffic. Thrift, Native tramsport is disabled. We are running upgradesstables on this node to see whether the newer version can re-write all the sstables and somehow get us out of this problem. was (Author: ferozshaik...@gmail.com): Schema is attached - [^schema.txt] ^We current use CQL . Legacy mode was thrift.^ This cluster is one of the oldest and very first Cassandra implementations. so pretty legacy data in there.We also witness that it has tombstones for certain tables . We see them logged in the system.log. Could it be this specific behavior of this legacy data combined with tombstones that causing the problem? Currently, we have removed that node from serving traffic. Thrift, Native tramsport is disabled. We are running upgradesstables on this node to see whether the newer version can re-write all the sstables and somehow get us out of this problem. > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: schema.txt, stack_trace.txt > > > We have also hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901760#comment-16901760 ] feroz shaik edited comment on CASSANDRA-15263 at 8/7/19 6:22 AM: - Schema is attached - [^schema.txt] ^We current use CQL . Legacy mode was thrift.^ This cluster is one of the oldest and very first Cassandra implementations. so pretty legacy data in there.We also witness that it has tombstones for certain tables . We see them logged in the system.log. Could it be this specific behavior of this legacy data combined with tombstones that causing the problem? Currently, we have removed that node from serving traffic. Thrift, Native tramsport is disabled. We are running upgradesstables on this node to see whether the newer version can re-write all the sstables and somehow get us out of this problem. was (Author: ferozshaik...@gmail.com): Schema is attached - [^schema.txt] ^We current use CQL . Legacy mode was thrift.^ > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: schema.txt, stack_trace.txt > > > We have also hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901760#comment-16901760 ] feroz shaik edited comment on CASSANDRA-15263 at 8/7/19 6:18 AM: - Schema is attached - [^schema.txt] ^We current use CQL . Legacy mode was thrift.^ was (Author: ferozshaik...@gmail.com): Schema is attached - [^schema.txt] > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: schema.txt, stack_trace.txt > > > We have also hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901760#comment-16901760 ] feroz shaik commented on CASSANDRA-15263: - Schema is attached - [^schema.txt] > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: schema.txt, stack_trace.txt > > > We have also hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[ https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feroz shaik updated CASSANDRA-15263: Attachment: schema.txt > LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null > --- > > Key: CASSANDRA-15263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15263 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: feroz shaik >Priority: Normal > Labels: 2.1.16, 3.11.4 > Attachments: schema.txt, stack_trace.txt > > > We have also hit a problem today while upgrading from 2.1.16 to 3.11.4. > we encountered this as soon as the first node started up with 3.11.4 > The full error stack is attached - [^stack_trace.txt] > > The below errors continued in the log file as long as the process was up. > ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778 > ErrorMessage.java:384 - Unexpected exception during request > java.lang.NullPointerException: null > ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454 > > The nodetool version says 3.11.4 and the no of connections on native por t- > 9042 was similar to other nodes. The exceptions were scary that we had to > call off the change. Any help and insights to this problem from the community > is appreciated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
feroz shaik updated CASSANDRA-15263:
Description:
We have also hit a problem today while upgrading from 2.1.16 to 3.11.4.
we encountered this as soon as the first node started up with 3.11.4
The full error stack is attached - [^stack_trace.txt]
The below errors continued in the log file as long as the process was up.
ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
The nodetool version says 3.11.4 and the no of connections on native por t-
9042 was similar to other nodes. The exceptions were scary that we had to call
off the change. Any help and insights to this problem from the community is
appreciated.
was:
We have also hit a problem today while upgrading from 2.1.16 to 3.11.4.
we encountered this as soon as the first node started up with 3.11.4
The full error stack is attached - [^stack_trace.txt]
The below errors continued in the log file as long as the process was up.
ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-13] 2019-08-06 03:01:29,983
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-13,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:01:31,119
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-6] 2019-08-06 03:01:46,262
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-15] 2019-08-06 03:01:46,520
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-2] 2019-08-06 03:01:48,842
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-2,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-1] 2019-08-06 03:01:50,351
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-5] 2019-08-06 03:02:06,061
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-8] 2019-08-06 03:02:07,616
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-8,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:02:08,384
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-5] 2019-08-06 03:02:10,244
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
The nodetool version says 3.11.4 and the no of connections on native por t-
9042 was similar to other nodes. The exceptions were scary that we had to call
off the change. Any help and insights to this problem from the community is
appreciated.
> LegacyLayout
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
feroz shaik updated CASSANDRA-15263:
Description:
We have also hit a problem today while upgrading from 2.1.16 to 3.11.4.
we encountered this as soon as the first node started up with 3.11.4
The full error stack is attached - [^stack_trace.txt]
The below errors continued in the log file as long as the process was up.
ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-13] 2019-08-06 03:01:29,983
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-13,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:01:31,119
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-6] 2019-08-06 03:01:46,262
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-15] 2019-08-06 03:01:46,520
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-2] 2019-08-06 03:01:48,842
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-2,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-1] 2019-08-06 03:01:50,351
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-5] 2019-08-06 03:02:06,061
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-8] 2019-08-06 03:02:07,616
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-8,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:02:08,384
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-5] 2019-08-06 03:02:10,244
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
The nodetool version says 3.11.4 and the no of connections on native por t-
9042 was similar to other nodes. The exceptions were scary that we had to call
off the change. Any help and insights to this problem from the community is
appreciated.
was:
We have also hit a problem today while upgrading from 2.1.16 to 3.11.4.
we encountered this as soon as the first node started up with 3.11.4
The full error stack is as below:
WARN [ReadStage-4] 2019-08-06 02:57:57,408
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-4,5,main]: {}
java.lang.NullPointerException: null
at
org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.updateDigest(LegacyLayout.java:2433)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.LegacyLayout$LegacyUnfilteredPartition.digest(LegacyLayout.java:1479)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:182)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:263)
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
feroz shaik updated CASSANDRA-15263:
Attachment: stack_trace.txt
> LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
> ---
>
> Key: CASSANDRA-15263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15263
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: feroz shaik
>Priority: Normal
> Labels: 2.1.16, 3.11.4
> Attachments: stack_trace.txt
>
>
> We have also hit a problem today while upgrading from 2.1.16 to 3.11.4.
> we encountered this as soon as the first node started up with 3.11.4
> The full error stack is as below:
> WARN [ReadStage-4] 2019-08-06 02:57:57,408
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-4,5,main]: {}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.updateDigest(LegacyLayout.java:2433)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.LegacyLayout$LegacyUnfilteredPartition.digest(LegacyLayout.java:1479)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:182)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:263)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:140)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadResponse.createDigestResponse(ReadResponse.java:87)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_131]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
>
>
> The below errors continued in the log file as long as the process was up.
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> WARN [ReadStage-13] 2019-08-06 03:01:29,983
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-13,5,main]: {}
>
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
feroz shaik updated CASSANDRA-15263:
Impacts: (was: None)
> LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
> ---
>
> Key: CASSANDRA-15263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15263
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: feroz shaik
>Priority: Normal
> Labels: 2.1.16, 3.11.4
>
> We have also hit a problem today while upgrading from 2.1.16 to 3.11.4.
> we encountered this as soon as the first node started up with 3.11.4
> The full error stack is as below:
> WARN [ReadStage-4] 2019-08-06 02:57:57,408
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-4,5,main]: {}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.updateDigest(LegacyLayout.java:2433)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.LegacyLayout$LegacyUnfilteredPartition.digest(LegacyLayout.java:1479)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:182)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:263)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:140)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadResponse.createDigestResponse(ReadResponse.java:87)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_131]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
>
>
> The below errors continued in the log file as long as the process was up.
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> WARN [ReadStage-13] 2019-08-06 03:01:29,983
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-13,5,main]: {}
> java.lang.NullPointerException: null
> ERROR
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
feroz shaik updated CASSANDRA-15263:
Labels: 2.1.16 3.11.4 (was: 2.1.16, 3.11.4)
> LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
> ---
>
> Key: CASSANDRA-15263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15263
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: feroz shaik
>Priority: Normal
> Labels: 2.1.16, 3.11.4
>
> We have also hit a problem today while upgrading from 2.1.16 to 3.11.4.
> we encountered this as soon as the first node started up with 3.11.4
> The full error stack is as below:
> WARN [ReadStage-4] 2019-08-06 02:57:57,408
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-4,5,main]: {}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.updateDigest(LegacyLayout.java:2433)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.LegacyLayout$LegacyUnfilteredPartition.digest(LegacyLayout.java:1479)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:182)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:263)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:140)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadResponse.createDigestResponse(ReadResponse.java:87)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_131]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
>
>
> The below errors continued in the log file as long as the process was up.
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> WARN [ReadStage-13] 2019-08-06 03:01:29,983
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-13,5,main]: {}
> java.lang.NullPointerException:
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
feroz shaik updated CASSANDRA-15263:
Platform: (was: All)
> LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
> ---
>
> Key: CASSANDRA-15263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15263
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: feroz shaik
>Priority: Normal
>
> We have also hit a problem today while upgrading from 2.1.16 to 3.11.4.
> we encountered this as soon as the first node started up with 3.11.4
> The full error stack is as below:
> WARN [ReadStage-4] 2019-08-06 02:57:57,408
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-4,5,main]: {}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.updateDigest(LegacyLayout.java:2433)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.LegacyLayout$LegacyUnfilteredPartition.digest(LegacyLayout.java:1479)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:182)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:263)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:140)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadResponse.createDigestResponse(ReadResponse.java:87)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_131]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
>
>
> The below errors continued in the log file as long as the process was up.
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> WARN [ReadStage-13] 2019-08-06 03:01:29,983
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-13,5,main]: {}
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-2] 2019-08-06
[jira] [Updated] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
feroz shaik updated CASSANDRA-15263:
Labels: 2.1.16, 3.11.4 (was: )
> LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
> ---
>
> Key: CASSANDRA-15263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15263
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: feroz shaik
>Priority: Normal
> Labels: 2.1.16,, 3.11.4
>
> We have also hit a problem today while upgrading from 2.1.16 to 3.11.4.
> we encountered this as soon as the first node started up with 3.11.4
> The full error stack is as below:
> WARN [ReadStage-4] 2019-08-06 02:57:57,408
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-4,5,main]: {}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.updateDigest(LegacyLayout.java:2433)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.LegacyLayout$LegacyUnfilteredPartition.digest(LegacyLayout.java:1479)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:182)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:263)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:140)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadResponse.createDigestResponse(ReadResponse.java:87)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_131]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
>
>
> The below errors continued in the log file as long as the process was up.
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> WARN [ReadStage-13] 2019-08-06 03:01:29,983
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-13,5,main]: {}
> java.lang.NullPointerException: null
> ERROR
[jira] [Created] (CASSANDRA-15263) LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
feroz shaik created CASSANDRA-15263:
---
Summary: LegacyLayout RangeTombstoneList throws
java.lang.NullPointerException: null
Key: CASSANDRA-15263
URL: https://issues.apache.org/jira/browse/CASSANDRA-15263
Project: Cassandra
Issue Type: Bug
Components: Cluster/Schema
Reporter: feroz shaik
We have also hit a problem today while upgrading from 2.1.16 to 3.11.4.
we encountered this as soon as the first node started up with 3.11.4
The full error stack is as below:
WARN [ReadStage-4] 2019-08-06 02:57:57,408
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-4,5,main]: {}
java.lang.NullPointerException: null
at
org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.updateDigest(LegacyLayout.java:2433)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.LegacyLayout$LegacyUnfilteredPartition.digest(LegacyLayout.java:1479)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:182)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:263)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:140)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ReadResponse.createDigestResponse(ReadResponse.java:87)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
[apache-cassandra-3.11.4.jar:3.11.4]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
The below errors continued in the log file as long as the process was up.
ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-13] 2019-08-06 03:01:29,983
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-13,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:01:31,119
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-6] 2019-08-06 03:01:46,262
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-15] 2019-08-06 03:01:46,520
ErrorMessage.java:384 - Unexpected exception during request
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901049#comment-16901049
]
feroz shaik commented on CASSANDRA-15172:
-
Sure, I shall raise a separate request.
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> at
>
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900882#comment-16900882
]
feroz shaik commented on CASSANDRA-15172:
-
We have currently isolated this node (cut off thrift , native protocols) and
trying to run upgradesstables to see if it can re-write all the files and stop
logging the below message.
"WARN [ReadStage-6] 2019-08-06 10:44:09,773
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-6,5,main]: {}
java.lang.NullPointerException: null"
I will keep the thread posted about its outcome,
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
>
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900879#comment-16900879
]
feroz shaik commented on CASSANDRA-15172:
-
What we also know is that the nodes suffered some heavy tombstones recently..
could it be under specific condition like "tombstones" plus reading legacy
version files is hitting this problem. [~Sagges] did you cluster have any
tombstone references at all?
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
>
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900876#comment-16900876
]
feroz shaik commented on CASSANDRA-15172:
-
Full stack trace is as below:
WARN [ReadStage-4] 2019-08-06 02:57:57,408
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-4,5,main]: {}
java.lang.NullPointerException: null
at
org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.updateDigest(LegacyLayout.java:2433)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.LegacyLayout$LegacyUnfilteredPartition.digest(LegacyLayout.java:1479)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:182)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:263)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:140)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ReadResponse.createDigestResponse(ReadResponse.java:87)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-3.11.4.jar:3.11.4]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
[apache-cassandra-3.11.4.jar:3.11.4]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
>
[jira] [Comment Edited] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900581#comment-16900581
]
feroz shaik edited comment on CASSANDRA-15172 at 8/6/19 3:37 AM:
-
We have also hit this problem today while upgrading from 2.1.16 to 3.11.4/
we encountered this as soon as node started up with 3.11.4
WARN [ReadStage-4] 2019-08-06 02:57:57,408
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-4,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-32] 2019-08-06 02:14:20,353
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
and the below errors continued in the logfile as long as the process was up.
ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-13] 2019-08-06 03:01:29,983
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-13,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:01:31,119
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-6] 2019-08-06 03:01:46,262
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-15] 2019-08-06 03:01:46,520
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-2] 2019-08-06 03:01:48,842
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-2,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-1] 2019-08-06 03:01:50,351
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-5] 2019-08-06 03:02:06,061
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-8] 2019-08-06 03:02:07,616
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-8,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:02:08,384
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-5] 2019-08-06 03:02:10,244
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
The nodetool version says 3.11.4 and the no of connections on 9042 was similar
to other nodes. The exceptions were scary that we had to call off the change.
Any help and insights to this problem from the community is appreciated.
was (Author: ferozshaik...@gmail.com):
We have also hit this problem today while upgrading from 2.1.16 to 3.11.4/
we encountered this as soon as node started up with 3.11.4
ERROR [Native-Transport-Requests-32] 2019-08-06 02:14:20,353
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
and the below errors continued in the logfile as long as the process was up.
ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900581#comment-16900581
]
feroz shaik commented on CASSANDRA-15172:
-
We have also hit this problem today while upgrading from 2.1.16 to 3.11.4/
we encountered this as soon as node started up with 3.11.4
ERROR [Native-Transport-Requests-32] 2019-08-06 02:14:20,353
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
and the below errors continued in the logfile as long as the process was up.
ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-13] 2019-08-06 03:01:29,983
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-13,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:01:31,119
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-6] 2019-08-06 03:01:46,262
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-15] 2019-08-06 03:01:46,520
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-2] 2019-08-06 03:01:48,842
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-2,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-1] 2019-08-06 03:01:50,351
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-5] 2019-08-06 03:02:06,061
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-8] 2019-08-06 03:02:07,616
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-8,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:02:08,384
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-5] 2019-08-06 03:02:10,244
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
The nodetool version says 3.11.4 and the no of connections on 9042 was similar
to other nodes. The exceptions were scary that we had to call off the change.
Any help and insights to this problem from the community is appreciated.
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036
