Re: Node removal causes spike in pending native-transport requests and clients suffer

[Gil Ganz]

Hey Bowen
I agree it's better to have smaller servers in general, this is the smaller
servers version :)
In this case, I wouldn't say the data model is bad, and we certainly do our
best to tune everything so less hardware is needed.
It's just that the data and amount of requests/s is very big to begin with,
multiple datacenters around the world (on-prem), with each datacenter
having close to 100 servers.
Making the servers smaller would mean a very large cluster, which has other
implications when it's on-prem.


On Fri, Mar 12, 2021 at 1:30 AM Bowen Song  wrote:

> May I ask why do you scale your Cassandra cluster vertically instead of
> horizontally as recommended?
>
> I'm asking because I had dealt with a vertically scaled cluster before. It
> was because they had query performance issue and blamed the hardware wasn't
> strong enough. Scaling vertically had helped them to improve the query
> performance, but it turned out the root caused was bad data modelling, and
> it gradually got worse with the ever increasing data size. Eventually they
> reached the roof of what money can realistically buy - 256GB RAM and 16
> cores 3.x GHz CPU per server in their case.
>
> Is that your case too? Bigger RAM, more cores and higher CPU frequency to
> help "fix" the performance issue? I really hope not.
>
>
> On 11/03/2021 09:57, Gil Ganz wrote:
>
> Yes. 192gb.
>
> On Thu, Mar 11, 2021 at 10:29 AM Kane Wilson  
> wrote:
>
>> That is a very large heap.  I presume you are using G1GC? How much memory
>> do your servers have?
>>
>> raft.so - Cassandra consulting, support, managed services
>>
>> On Thu., 11 Mar. 2021, 18:29 Gil Ganz,  wrote:
>>
>>> I always prefer to do decommission, but the issue here  is these servers
>>> are on-prem, and disks die from time to time.
>>> It's a very large cluster, in multiple datacenters around the world, so
>>> it can take some time before we have a replacement, so we usually need to
>>> run removenode in such cases.
>>>
>>> Other than that there are no issues in the cluster, the load is
>>> reasonable, and when this issue happens, following a removenode, this huge
>>> number of NTR is what I see, weird thing it's only on some nodes.
>>> I have been running with a very small
>>> native_transport_max_concurrent_requests_in_bytes  setting for a few days
>>> now on some nodes (few mb's compared to the default 0.8 of a 60gb heap), it
>>> looks like it's good enough for the app, will roll it out to the entire dc
>>> and test removal again.
>>>
>>>
>>> On Tue, Mar 9, 2021 at 10:51 AM Kane Wilson  
>>> wrote:
>>>
 It's unlikely to help in this case, but you should be using nodetool
 decommission on the node you want to remove rather than removenode from
 another node (and definitely don't force removal)

 native_transport_max_concurrent_requests_in_bytes defaults to 10% of
 the heap, which I suppose depending on your configuration could potentially
 result in a smaller number of concurrent requests than previously. It's
 worth a shot setting it higher to see if the issue is related. Is this the
 only issue you see on the cluster? I assume load on the cluster is still
 low/reasonable and the only symptom you're seeing is the increased NTR
 requests?

 raft.so - Cassandra consulting, support, and managed services


 On Mon, Mar 8, 2021 at 10:47 PM Gil Ganz  wrote:

>
> Hey,
> We have a 3.11.9 cluster (recently upgraded from 2.1.14), and after
> the upgrade we have an issue when we remove a node.
>
> The moment I run the removenode command, 3 servers in the same dc
> start to have a high amount of pending native-transport-requests (getting
> to around 1M) and clients are having issues due to that. We are using
> vnodes (32), so I I don't see why I would have 3 servers busier than 
> others
> (RF is 3 but I don't see why it will be related).
>
> Each node has a few TB of data, and in the past we were able to remove
> a node in ~half a day, today what happens is in the first 1-2 hours we 
> have
> these issues with some nodes, then things go quite, remove is still 
> running
> and clients are ok, a few hours later the same issue is back (with same
> nodes as the problematic ones), and clients have issues again, leading us
> to run removenode force.
>
> Reducing stream throughput and number of compactors has helped
> to mitigate the issues a bit, but we still have this issue of pending
> native-transport requests getting to insane numbers and clients suffering,
> eventually causing us to run remove force. Any idea?
>
> I saw since 3.11.6 there is a parameter
> native_transport_max_concurrent_requests_in_bytes, looking into setting
> this, perhaps this will prevent the amount of pending tasks to get so 
> high.
>
> Gil
>

Re: What Happened To Alternate Storage And Rocksandra?

[Jeff Jirsa]

As someone who watched some of the work (but wasn’t really involved), I think a 
bunch of it fizzled for various reasons

The rocks stuff was built (mostly? Completely?) by one company for their use 
case (the best kind of open source), but wasn’t in a form that was easy to 
commit upstream - the work to make clean abstractions to make it pluggable ran 
up against other priorities and ultimately slowed to a halt.

Some of the abstractions are patch available or committed, but I’m not sure if 
the rocksandra folks are likely to continue rebasing and trying to get it 
finished / upstreamed post 4.0

That said, there are indeed a ton of things in rocks I hope we adopt. 

> On Mar 12, 2021, at 10:50 AM, Gareth Collins  
> wrote:
> 
> 
> Hi,
> 
> I remember a couple of years ago there was some noise about Rocksandra 
> (Cassandra using rocksdb for storage) and opening up Cassandra to alternate 
> storage mechanisms.
> 
> I haven't seen anything about it for a while now though. The last commit to 
> Rocksandra on github was in Nov 2019. The associated JIRA items 
> (CASSANDRA-13474 and CASSANDRA-13476) haven't had any activity since 2019 
> either.
> 
> I was wondering whether anyone knew anything about it. Was it decided that 
> this wasn't a good idea after all (the alleged performance differences 
> weren't worth it...or were exaggerated)? Or is it just that it still may be a 
> good idea, but there are no resources available to make this happen (e.g. 
> perhaps the original sponsor moved onto other things)?
> 
> I ask because I was looking at RocksDB/Kafka Streams for another project 
> (which may replace some functionality which currently uses Cassandra)...and 
> was wondering if there could be some important info about RocksDB I may be 
> missing.
> 
> thanks in advance,
> Gareth Collins

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: What Happened To Alternate Storage And Rocksandra?

[onmstester onmstester]

Beside the enhancements at storage layer, i think there are couple of good 
ideas in Rocksdb that could be used in Cassandra, like the one with disabling 
sort at memtable-insert part (write data fast like commitlig) and only sort the 
data when flushing/creating sst files.

Sent using https://www.zoho.com/mail/






 On Fri, 12 Mar 2021 23:47:05 +0330 Elliott Sims  
wrote 


I'm not too familiar with the details on what's happened more recently, but I 
do remember that while Rocksandra was very favorably compared to Cassandra 2.x, 
the improvements looked fairly similar in nature and magnitude to what 
Cassandra got from the move to the 3.x sstable format and increased use of 
off-heap memory.  That might have damped a lot of the enthusiasm for further 
development.


On Fri, Mar 12, 2021 at 10:50 AM Gareth Collins 
 wrote:

Hi,

I remember a couple of years ago there was some noise about Rocksandra 
(Cassandra using rocksdb for storage) and opening up Cassandra to alternate 
storage mechanisms.



I haven't seen anything about it for a while now though. The last commit to 
Rocksandra on github was in Nov 2019. The associated JIRA items 
(CASSANDRA-13474 and CASSANDRA-13476) haven't had any activity since 2019 
either.



I was wondering whether anyone knew anything about it. Was it decided that this 
wasn't a good idea after all (the alleged performance differences weren't worth 
it...or were exaggerated)? Or is it just that it still may be a good idea, but 
there are no resources available to make this happen (e.g. perhaps the original 
sponsor moved onto other things)?



I ask because I was looking at RocksDB/Kafka Streams for another project (which 
may replace some functionality which currently uses Cassandra)...and was 
wondering if there could be some important info about RocksDB I may be missing.



thanks in advance,

Gareth Collins

Re: No node was available to execute query error

[Bowen Song]

The partition size min/avg/max of 8409008/15096925/25109160 bytes looks 
fine for the table fieldcounts, but the number of partitions is a bit 
worrying. Only 3 partitions? Are you expecting the partition size 
(instead of number of partitions) to grow in the future? That can lead 
to a lots of headaches.


Forget about the fieldcounts table for now, the doc table looks really 
bad. It has min/avg/max partition size of 24602/7052951452/63771372175 
bytes, the partition sizes are severely unevenly distributed, and the 
over 60GB partition is way too big.


You really need to redesign your table schemas, and avoid creating large 
or uneven partitions.



On 12/03/2021 18:52, Joe Obernberger wrote:


Thank you very much for helping me out on this!  The table fieldcounts 
is currently pretty small - 6.4 million rows.


cfstats are:

Total number of tables: 81

Keyspace : doc
    Read Count: 3713134
    Read Latency: 0.2664131157130338 ms
    Write Count: 47513045
    Write Latency: 1.0725477948634947 ms
    Pending Flushes: 0
    Table: fieldcounts
    SSTable count: 3
    Space used (live): 16010248
    Space used (total): 16010248
    Space used by snapshots (total): 0
    Off heap memory used (total): 4947
    SSTable Compression Ratio: 0.3994304032360534
    Number of partitions (estimate): 3
    Memtable cell count: 0
    Memtable data size: 0
    Memtable off heap memory used: 0
    Memtable switch count: 0
    Local read count: 379
    Local read latency: NaN ms
    Local write count: 0
    Local write latency: NaN ms
    Pending flushes: 0
    Percent repaired: 100.0
    Bloom filter false positives: 0
    Bloom filter false ratio: 0.0
    Bloom filter space used: 48
    Bloom filter off heap memory used: 24
    Index summary off heap memory used: 51
    Compression metadata off heap memory used: 4872
    Compacted partition minimum bytes: 8409008
    Compacted partition maximum bytes: 25109160
    Compacted partition mean bytes: 15096925
    Average live cells per slice (last five minutes): NaN
    Maximum live cells per slice (last five minutes): 0
    Average tombstones per slice (last five minutes): NaN
    Maximum tombstones per slice (last five minutes): 0
    Dropped Mutations: 0

Commitlog is on a separate spindle on the 7 node cluster.  All disks 
are SATA (spinning rust as they say!).  This is an R platform, but I 
will switch to NetworkTopologyStrategy.  I'm using Prometheus and 
Grafana to monitor Cassandra and the CPU load is typically 100 to 200% 
on most of the nodes.  Disk IO is typically pretty low.


Performance - in general Async is about 10x faster.
ExecuteAsync:
35mSec for 364 rows.
8120mSec for 205001 rows.
14788mSec for 345001 rows.
4117mSec for 86400 rows.

23,330 rows per second on average

Execute:
232mSec for 364 rows.
584869mSec for 1263283 rows
46290mSec for 86400 rows

2,160 rows per second on average

Curious - our largest table (doc) has the following stats - is it not 
partitioned well?


Total number of tables: 81

Keyspace : doc
    Read Count: 3713134
    Read Latency: 0.2664131157130338 ms
    Write Count: 47513045
    Write Latency: 1.0725477948634947 ms
    Pending Flushes: 0
    Table: doc
    SSTable count: 26
    Space used (live): 57124641753
    Space used (total): 57124641753
    Space used by snapshots (total): 113012646218
    Off heap memory used (total): 27331913
    SSTable Compression Ratio: 0.2531585373184219
    Number of partitions (estimate): 12
    Memtable cell count: 0
    Memtable data size: 0
    Memtable off heap memory used: 0
    Memtable switch count: 0
    Local read count: 27169
    Local read latency: NaN ms
    Local write count: 0
    Local write latency: NaN ms
    Pending flushes: 0
    Percent repaired: 0.0
    Bloom filter false positives: 0
    Bloom filter false ratio: 0.0
    Bloom filter space used: 576
    Bloom filter off heap memory used: 368
    Index summary off heap memory used: 425
    Compression metadata off heap memory used: 27331120
    Compacted partition minimum bytes: 24602
    Compacted partition maximum bytes: 63771372175
    Compacted partition mean bytes: 7052951452
    Average live cells per slice (last five minutes): NaN
 

Re: What Happened To Alternate Storage And Rocksandra?

[Elliott Sims]

I'm not too familiar with the details on what's happened more recently, but
I do remember that while Rocksandra was very favorably compared to
Cassandra 2.x, the improvements looked fairly similar in nature and
magnitude to what Cassandra got from the move to the 3.x sstable format and
increased use of off-heap memory.  That might have damped a lot of the
enthusiasm for further development.

On Fri, Mar 12, 2021 at 10:50 AM Gareth Collins 
wrote:

> Hi,
>
> I remember a couple of years ago there was some noise about Rocksandra
> (Cassandra using rocksdb for storage) and opening up Cassandra to alternate
> storage mechanisms.
>
> I haven't seen anything about it for a while now though. The last commit
> to Rocksandra on github was in Nov 2019. The associated JIRA items
> (CASSANDRA-13474 and CASSANDRA-13476) haven't had any activity since 2019
> either.
>
> I was wondering whether anyone knew anything about it. Was it decided that
> this wasn't a good idea after all (the alleged performance differences
> weren't worth it...or were exaggerated)? Or is it just that it still may be
> a good idea, but there are no resources available to make this happen (e.g.
> perhaps the original sponsor moved onto other things)?
>
> I ask because I was looking at RocksDB/Kafka Streams for another project
> (which may replace some functionality which currently uses Cassandra)...and
> was wondering if there could be some important info about RocksDB I may be
> missing.
>
> thanks in advance,
> Gareth Collins
>

Re: No node was available to execute query error

[Joe Obernberger]

Thank you very much for helping me out on this!  The table fieldcounts 
is currently pretty small - 6.4 million rows.


cfstats are:

Total number of tables: 81

Keyspace : doc
        Read Count: 3713134
        Read Latency: 0.2664131157130338 ms
        Write Count: 47513045
        Write Latency: 1.0725477948634947 ms
        Pending Flushes: 0
                Table: fieldcounts
                SSTable count: 3
                Space used (live): 16010248
                Space used (total): 16010248
                Space used by snapshots (total): 0
                Off heap memory used (total): 4947
                SSTable Compression Ratio: 0.3994304032360534
                Number of partitions (estimate): 3
                Memtable cell count: 0
                Memtable data size: 0
                Memtable off heap memory used: 0
                Memtable switch count: 0
                Local read count: 379
                Local read latency: NaN ms
                Local write count: 0
                Local write latency: NaN ms
                Pending flushes: 0
                Percent repaired: 100.0
                Bloom filter false positives: 0
                Bloom filter false ratio: 0.0
                Bloom filter space used: 48
                Bloom filter off heap memory used: 24
                Index summary off heap memory used: 51
                Compression metadata off heap memory 
used: 4872

                Compacted partition minimum bytes: 8409008
                Compacted partition maximum bytes: 25109160
                Compacted partition mean bytes: 15096925
                Average live cells per slice (last five 
minutes): NaN
                Maximum live cells per slice (last five 
minutes): 0
                Average tombstones per slice (last five 
minutes): NaN
                Maximum tombstones per slice (last five 
minutes): 0

                Dropped Mutations: 0

Commitlog is on a separate spindle on the 7 node cluster.  All disks 
are SATA (spinning rust as they say!).  This is an R platform, but I 
will switch to NetworkTopologyStrategy.  I'm using Prometheus and 
Grafana to monitor Cassandra and the CPU load is typically 100 to 200% 
on most of the nodes.  Disk IO is typically pretty low.


Performance - in general Async is about 10x faster.
ExecuteAsync:
35mSec for 364 rows.
8120mSec for 205001 rows.
14788mSec for 345001 rows.
4117mSec for 86400 rows.

23,330 rows per second on average

Execute:
232mSec for 364 rows.
584869mSec for 1263283 rows
46290mSec for 86400 rows

2,160 rows per second on average

Curious - our largest table (doc) has the following stats - is it not 
partitioned well?


Total number of tables: 81

Keyspace : doc
        Read Count: 3713134
        Read Latency: 0.2664131157130338 ms
        Write Count: 47513045
        Write Latency: 1.0725477948634947 ms
        Pending Flushes: 0
                Table: doc
                SSTable count: 26
                Space used (live): 57124641753
                Space used (total): 57124641753
                Space used by snapshots (total): 113012646218
                Off heap memory used (total): 27331913
                SSTable Compression Ratio: 0.2531585373184219
                Number of partitions (estimate): 12
                Memtable cell count: 0
                Memtable data size: 0
                Memtable off heap memory used: 0
                Memtable switch count: 0
                Local read count: 27169
                Local read latency: NaN ms
                Local write count: 0
                Local write latency: NaN ms
                Pending flushes: 0
                Percent repaired: 0.0
                Bloom filter false positives: 0
                Bloom filter false ratio: 0.0
                Bloom filter space used: 576
                Bloom filter off heap memory used: 368
                Index summary off heap memory used: 425
                Compression metadata off heap memory 
used: 27331120

                

What Happened To Alternate Storage And Rocksandra?

[Gareth Collins]

Hi,

I remember a couple of years ago there was some noise about Rocksandra
(Cassandra using rocksdb for storage) and opening up Cassandra to alternate
storage mechanisms.

I haven't seen anything about it for a while now though. The last commit to
Rocksandra on github was in Nov 2019. The associated JIRA items
(CASSANDRA-13474 and CASSANDRA-13476) haven't had any activity since 2019
either.

I was wondering whether anyone knew anything about it. Was it decided that
this wasn't a good idea after all (the alleged performance differences
weren't worth it...or were exaggerated)? Or is it just that it still may be
a good idea, but there are no resources available to make this happen (e.g.
perhaps the original sponsor moved onto other things)?

I ask because I was looking at RocksDB/Kafka Streams for another project
(which may replace some functionality which currently uses Cassandra)...and
was wondering if there could be some important info about RocksDB I may be
missing.

thanks in advance,
Gareth Collins

Re: No node was available to execute query error

[Bowen Song]

The highlight is "millions rows in a **single** query". Fetching that 
amount of data in a single query is bad, because the Java heap memory 
overhead. You can fetch millions of rows in Cassandra, just make sure 
you do that over thousands or millions of queries, not one single query.



On 12/03/2021 15:32, Joe Obernberger wrote:


One question on the 'millions rows in a single query'.  How would you 
process that many rows?  At some point, I'd like to be able to process 
10-100 billion rows.  Isn't that something that can be done with 
Cassandra?  I'm coming from HBase where we'd run map reduce jobs.

Thank you.

-Joe

On 3/12/2021 9:07 AM, Bowen Song wrote:


Millions rows in a single query? That sounds like a bad idea to me. 
Your "NoNodeAvailableException" could be caused by stop-the-world GC 
pauses, and the GC pauses are likely caused by the query itself.


On 12/03/2021 13:39, Joe Obernberger wrote:


Thank you Paul and Erick.  The keyspace is defined like this:
CREATE KEYSPACE doc WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '3'}  AND durable_writes = true;


Would that cause this?

The program that is having the problem selects data, calculates 
stuff, and inserts.  It works with smaller selects, but when the 
number of rows is in the millions, I start to get this error.  Since 
it works with smaller sets, I don't believe it to be a network 
error.  All the nodes are definitely up as other processes are 
working OK, it's just this one program that fails.


The full stack trace:

Error: com.datastax.oss.driver.api.core.NoNodeAvailableException: No 
node was available to execute the query
com.datastax.oss.driver.api.core.NoNodeAvailableException: No node 
was available to execute the query
    at 
com.datastax.oss.driver.api.core.NoNodeAvailableException.copy(NoNodeAvailableException.java:40)
    at 
com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
    at 
com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
    at 
com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
    at 
com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
    at 
com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
    at 
com.abc..fieldanalyzer.FTAProcess.udpateCassandraFTAMetrics(FTAProcess.java:275)
    at 
com.abc..fieldanalyzer.FTAProcess.storeResults(FTAProcess.java:216)
    at 
com.abc..fieldanalyzer.FTAProcess.startProcess(FTAProcess.java:199)

    at com.abc..fieldanalyzer.Main.main(Main.java:20)

FTAProcess like 275 is:

ResultSet rs = session.execute(getFieldCounts.bind().setString(0, 
rb.getSource()).setString(1, rb.getFieldName()));


-Joe

On 3/12/2021 8:30 AM, Paul Chandler wrote:

Hi Joe

This could also be caused by the replication factor of the 
keyspace, if you have NetworkTopologyStrategy and it doesn’t list a 
replication factor for the datacenter datacenter1 then you will get 
this error message too.


Paul

On 12 Mar 2021, at 13:07, Erick Ramirez 
mailto:erick.rami...@datastax.com>> 
wrote:


Does it get returned by the driver every single time? The 
NoNodeAvailableExceptiongets thrown when (1) all nodes are down, 
or (2) all the contact points are invalid from the driver's 
perspective.


Is it possible there's no route/connectivity from your app 
server(s) to the 172.16.x.xnetwork? If you post the full error 
message + full stacktrace, it might provide clues. Cheers!



 
	Virus-free. www.avg.com 
 



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: No node was available to execute query error

[Bowen Song]

Sleep-then-retry works is just another indicator that it's likely a GC 
pause related issue. I'd recommend you to check your Cassandra servers' 
GC logs first.


Do you know what's the maximum partition size for the doc.fieldcounts 
table? (Try the "nodetool cfstats doc.fieldcounts" command) I suspect 
this table has large partitions, which usually leads to GC issues.


As of your failed executeAsync() insert issue, do you know how many 
concurrent on-the-fly queries do you have? Cassandra driver has 
limitations on it, and new executeAsync() calls will fail when the limit 
is reached.


I'm also a bit concerned about your "significantly" slower inserts. 
Inserts (excluding "INSERT IF NOT EXISTS") should be very fast in 
Cassandra. How slow are they? Are they always slow like that, or usually 
fast but some are much slower than others? What does the CPU usage & 
disk IO look like on the Cassandra server? Do you have commitlog on the 
same disk as the data? Is it a spinning disk, SATA SSD or NVMe?


BTW, you really shouldn't use SimpleStrategy for production environments.


On 12/03/2021 15:18, Joe Obernberger wrote:


The queries that are failing are:

select fieldvalue, count from doc.ordered_fieldcounts where source=? 
and fieldname=? limit 10


Created with:
CREATE TABLE doc.ordered_fieldcounts (
    source text,
    fieldname text,
    count bigint,
    fieldvalue text,
    PRIMARY KEY ((source, fieldname), count, fieldvalue)
) WITH CLUSTERING ORDER BY (count DESC, fieldvalue ASC)

and:

select fieldvalue, count from doc.fieldcounts where source=? and 
fieldname=?


Created with:
CREATE TABLE doc.fieldcounts (
    source text,
    fieldname text,
    fieldvalue text,
    count bigint,
    PRIMARY KEY (source, fieldname, fieldvalue)
)

This really seems like a driver issue.  I put retry logic around the 
calls and now those queries work.  Basically if it throws an 
exception, I Thread.sleep(500) and then retry.  This seems to be a 
continuing theme with Cassandra in general.  Is this common practice?


After doing this retry logic, an insert statement started failing with 
an illegal state exception when I retried it (which makes sense).  
This insert was using session.executeAsync(boundStatement).  I changed 
that to just execute (instead of async) and now I get no errors, no 
retries anywhere.  The insert is *significantly* slower when running 
execute vs executeAsync.  When using executeAsync:


com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was 
available to execute the query
    at 
com.datastax.oss.driver.api.core.NoNodeAvailableException.copy(NoNodeAvailableException.java:40)
    at 
com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
    at 
com.datastax.oss.driver.internal.core.cql.MultiPageResultSet$RowIterator.maybeMoveToNextPage(MultiPageResultSet.java:99)
    at 
com.datastax.oss.driver.internal.core.cql.MultiPageResultSet$RowIterator.computeNext(MultiPageResultSet.java:91)
    at 
com.datastax.oss.driver.internal.core.cql.MultiPageResultSet$RowIterator.computeNext(MultiPageResultSet.java:79)
    at 
com.datastax.oss.driver.internal.core.util.CountingIterator.tryToComputeNext(CountingIterator.java:91)
    at 
com.datastax.oss.driver.internal.core.util.CountingIterator.hasNext(CountingIterator.java:86)
    at 
com.ngc.helios.fieldanalyzer.FTAProcess.handleOrderedFieldCounts(FTAProcess.java:684)
    at 
com.ngc.helios.fieldanalyzer.FTAProcess.storeResults(FTAProcess.java:214)
    at 
com.ngc.helios.fieldanalyzer.FTAProcess.startProcess(FTAProcess.java:190)

    at com.ngc.helios.fieldanalyzer.Main.main(Main.java:20)

The interesting part here is the the line that is now failing (line 
684 in FTAProcess) is:


if (itRs.hasNext())

where itRs is an iterator over a select query from another 
table.  I'm iterating over a result set from a select and inserting 
those results via executeAsync.


-Joe

On 3/12/2021 9:07 AM, Bowen Song wrote:


Millions rows in a single query? That sounds like a bad idea to me. 
Your "NoNodeAvailableException" could be caused by stop-the-world GC 
pauses, and the GC pauses are likely caused by the query itself.


On 12/03/2021 13:39, Joe Obernberger wrote:


Thank you Paul and Erick.  The keyspace is defined like this:
CREATE KEYSPACE doc WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '3'}  AND durable_writes = true;


Would that cause this?

The program that is having the problem selects data, calculates 
stuff, and inserts.  It works with smaller selects, but when the 
number of rows is in the millions, I start to get this error.  Since 
it works with smaller sets, I don't believe it to be a network 
error.  All the nodes are definitely up as other processes are 
working OK, it's just this one program that fails.


The full stack trace:

Error: com.datastax.oss.driver.api.core.NoNodeAvailableException: No 
node 

Re: No node was available to execute query error

[Joe Obernberger]

One question on the 'millions rows in a single query'.  How would you 
process that many rows?  At some point, I'd like to be able to process 
10-100 billion rows.  Isn't that something that can be done with 
Cassandra?  I'm coming from HBase where we'd run map reduce jobs.

Thank you.

-Joe

On 3/12/2021 9:07 AM, Bowen Song wrote:


Millions rows in a single query? That sounds like a bad idea to me. 
Your "NoNodeAvailableException" could be caused by stop-the-world GC 
pauses, and the GC pauses are likely caused by the query itself.


On 12/03/2021 13:39, Joe Obernberger wrote:


Thank you Paul and Erick.  The keyspace is defined like this:
CREATE KEYSPACE doc WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '3'}  AND durable_writes = true;


Would that cause this?

The program that is having the problem selects data, calculates 
stuff, and inserts.  It works with smaller selects, but when the 
number of rows is in the millions, I start to get this error.  Since 
it works with smaller sets, I don't believe it to be a network 
error.  All the nodes are definitely up as other processes are 
working OK, it's just this one program that fails.


The full stack trace:

Error: com.datastax.oss.driver.api.core.NoNodeAvailableException: No 
node was available to execute the query
com.datastax.oss.driver.api.core.NoNodeAvailableException: No node 
was available to execute the query
        at 
com.datastax.oss.driver.api.core.NoNodeAvailableException.copy(NoNodeAvailableException.java:40)
        at 
com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
        at 
com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
        at 
com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
        at 
com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
        at 
com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
        at 
com.abc..fieldanalyzer.FTAProcess.udpateCassandraFTAMetrics(FTAProcess.java:275)
        at 
com.abc..fieldanalyzer.FTAProcess.storeResults(FTAProcess.java:216)
        at 
com.abc..fieldanalyzer.FTAProcess.startProcess(FTAProcess.java:199)

        at com.abc..fieldanalyzer.Main.main(Main.java:20)

FTAProcess like 275 is:

ResultSet rs = session.execute(getFieldCounts.bind().setString(0, 
rb.getSource()).setString(1, rb.getFieldName()));


-Joe

On 3/12/2021 8:30 AM, Paul Chandler wrote:

Hi Joe

This could also be caused by the replication factor of the keyspace, 
if you have NetworkTopologyStrategy and it doesn’t list a 
replication factor for the datacenter datacenter1 then you will get 
this error message too.Â


Paul

On 12 Mar 2021, at 13:07, Erick Ramirez > wrote:


Does it get returned by the driver every single time? The 
NoNodeAvailableExceptiongets thrown when (1) all nodes are down, or 
(2) all the contact points are invalid from the driver's perspective.


Is it possible there's no route/connectivity from your app 
server(s) to the 172.16.x.xnetwork? If you post the full error 
message + full stacktrace, it might provide clues. Cheers!



 
	Virus-free. www.avg.com 
 



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: No node was available to execute query error

[Joe Obernberger]

The queries that are failing are:

select fieldvalue, count from doc.ordered_fieldcounts where source=? and 
fieldname=? limit 10


Created with:
CREATE TABLE doc.ordered_fieldcounts (
    source text,
    fieldname text,
    count bigint,
    fieldvalue text,
    PRIMARY KEY ((source, fieldname), count, fieldvalue)
) WITH CLUSTERING ORDER BY (count DESC, fieldvalue ASC)

and:

select fieldvalue, count from doc.fieldcounts where source=? and fieldname=?

Created with:
CREATE TABLE doc.fieldcounts (
    source text,
    fieldname text,
    fieldvalue text,
    count bigint,
    PRIMARY KEY (source, fieldname, fieldvalue)
)

This really seems like a driver issue.  I put retry logic around the 
calls and now those queries work.  Basically if it throws an exception, 
I Thread.sleep(500) and then retry.  This seems to be a continuing 
theme with Cassandra in general.  Is this common practice?


After doing this retry logic, an insert statement started failing with 
an illegal state exception when I retried it (which makes sense).  This 
insert was using session.executeAsync(boundStatement).  I changed that 
to just execute (instead of async) and now I get no errors, no retries 
anywhere.  The insert is *significantly* slower when running execute vs 
executeAsync.  When using executeAsync:


com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was 
available to execute the query
        at 
com.datastax.oss.driver.api.core.NoNodeAvailableException.copy(NoNodeAvailableException.java:40)
        at 
com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
        at 
com.datastax.oss.driver.internal.core.cql.MultiPageResultSet$RowIterator.maybeMoveToNextPage(MultiPageResultSet.java:99)
        at 
com.datastax.oss.driver.internal.core.cql.MultiPageResultSet$RowIterator.computeNext(MultiPageResultSet.java:91)
        at 
com.datastax.oss.driver.internal.core.cql.MultiPageResultSet$RowIterator.computeNext(MultiPageResultSet.java:79)
        at 
com.datastax.oss.driver.internal.core.util.CountingIterator.tryToComputeNext(CountingIterator.java:91)
        at 
com.datastax.oss.driver.internal.core.util.CountingIterator.hasNext(CountingIterator.java:86)
        at 
com.ngc.helios.fieldanalyzer.FTAProcess.handleOrderedFieldCounts(FTAProcess.java:684)
        at 
com.ngc.helios.fieldanalyzer.FTAProcess.storeResults(FTAProcess.java:214)
        at 
com.ngc.helios.fieldanalyzer.FTAProcess.startProcess(FTAProcess.java:190)

        at com.ngc.helios.fieldanalyzer.Main.main(Main.java:20)

The interesting part here is the the line that is now failing (line 684 
in FTAProcess) is:


if (itRs.hasNext())

where itRs is an iterator over a select query from another table.  
I'm iterating over a result set from a select and inserting those 
results via executeAsync.


-Joe

On 3/12/2021 9:07 AM, Bowen Song wrote:


Millions rows in a single query? That sounds like a bad idea to me. 
Your "NoNodeAvailableException" could be caused by stop-the-world GC 
pauses, and the GC pauses are likely caused by the query itself.


On 12/03/2021 13:39, Joe Obernberger wrote:


Thank you Paul and Erick.  The keyspace is defined like this:
CREATE KEYSPACE doc WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '3'}  AND durable_writes = true;


Would that cause this?

The program that is having the problem selects data, calculates 
stuff, and inserts.  It works with smaller selects, but when the 
number of rows is in the millions, I start to get this error.  Since 
it works with smaller sets, I don't believe it to be a network 
error.  All the nodes are definitely up as other processes are 
working OK, it's just this one program that fails.


The full stack trace:

Error: com.datastax.oss.driver.api.core.NoNodeAvailableException: No 
node was available to execute the query
com.datastax.oss.driver.api.core.NoNodeAvailableException: No node 
was available to execute the query
        at 
com.datastax.oss.driver.api.core.NoNodeAvailableException.copy(NoNodeAvailableException.java:40)
        at 
com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
        at 
com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
        at 
com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
        at 
com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
        at 
com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
        at 
com.abc..fieldanalyzer.FTAProcess.udpateCassandraFTAMetrics(FTAProcess.java:275)
        at 

Re: Barman equivalent for Cassandra?

[Jonathan Lacefield]

There is a community delivered tool named Medusa that may have what you're
looking for as well - https://cassandra.tools/medusa

Jonathan Lacefield
e. jlacefi...@datastax.com
w. www.datastax.com
schedule a meeting on my calendar



On Fri, Mar 12, 2021 at 8:07 AM David Tinker  wrote:

> Hi Guys
>
> I need to backup my 3 node Cassandra cluster to a remote machine. Is there
> a tool like Barman (really nice streaming backup tool for Postgresql) for
> Cassandra? Or does everyone roll their own scripts using snapshots and so
> on?
>
> The data is on all 3 nodes using about 900G of space on each.
>
> It would be difficult for me to recover even a day of lost data. An hour
> might be ok.
>
> Thanks
> David
>
>

Re: No node was available to execute query error

[Bowen Song]

Millions rows in a single query? That sounds like a bad idea to me. Your 
"NoNodeAvailableException" could be caused by stop-the-world GC pauses, 
and the GC pauses are likely caused by the query itself.


On 12/03/2021 13:39, Joe Obernberger wrote:


Thank you Paul and Erick.  The keyspace is defined like this:
CREATE KEYSPACE doc WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '3'}  AND durable_writes = true;


Would that cause this?

The program that is having the problem selects data, calculates stuff, 
and inserts.  It works with smaller selects, but when the number of 
rows is in the millions, I start to get this error. Since it works 
with smaller sets, I don't believe it to be a network error.  All the 
nodes are definitely up as other processes are working OK, it's just 
this one program that fails.


The full stack trace:

Error: com.datastax.oss.driver.api.core.NoNodeAvailableException: No 
node was available to execute the query
com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was 
available to execute the query
    at 
com.datastax.oss.driver.api.core.NoNodeAvailableException.copy(NoNodeAvailableException.java:40)
    at 
com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
    at 
com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
    at 
com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
    at 
com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
    at 
com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
    at 
com.abc..fieldanalyzer.FTAProcess.udpateCassandraFTAMetrics(FTAProcess.java:275)
    at 
com.abc..fieldanalyzer.FTAProcess.storeResults(FTAProcess.java:216)
    at 
com.abc..fieldanalyzer.FTAProcess.startProcess(FTAProcess.java:199)

    at com.abc..fieldanalyzer.Main.main(Main.java:20)

FTAProcess like 275 is:

ResultSet rs = session.execute(getFieldCounts.bind().setString(0, 
rb.getSource()).setString(1, rb.getFieldName()));


-Joe

On 3/12/2021 8:30 AM, Paul Chandler wrote:

Hi Joe

This could also be caused by the replication factor of the keyspace, 
if you have NetworkTopologyStrategy and it doesn’t list a replication 
factor for the datacenter datacenter1 then you will get this error 
message too.


Paul

On 12 Mar 2021, at 13:07, Erick Ramirez > wrote:


Does it get returned by the driver every single time? The 
NoNodeAvailableExceptiongets thrown when (1) all nodes are down, or 
(2) all the contact points are invalid from the driver's perspective.


Is it possible there's no route/connectivity from your app server(s) 
to the 172.16.x.xnetwork? If you post the full error message + full 
stacktrace, it might provide clues. Cheers!



 
	Virus-free. www.avg.com 
 



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: Barman equivalent for Cassandra?

[Erick Ramirez]

I'm not familiar with Barman but if you're looking for a backup software
for Cassandra, have a look at Medusa from The Last Pickle --
https://github.com/thelastpickle/cassandra-medusa/wiki.

It's open-source and is also used for https://k8ssandra.io/ -- the platform
for deploying Cassandra on Kubernetes with tools for repairs, backups and
monitoring built-in. Cheers!

Re: Barman equivalent for Cassandra?

[Bowen Song]

You can have a separate DC, so a physical destruction of an entire DC 
(such as a fire ) will not result in data loss; you can turn on 
automatic snapshot on truncate & drop table to help prevent some data 
losses caused by bugs and human errors; you can also have a cron job to 
take snapshots (and delete old snapshots), so you can rollback to an 
earlier version. In addition to storing snapshots off-site, you can also 
archive the commitlog files, even CDC files off-site to minimize the 
amount of data loss in case of a total physical destruction of the 
entire Cassandra cluster.


If you are okay with at most an hour of data loss, an hourly snapshot 
stored off-site should be more than enough. Note that SSTable files are 
immutable, and that makes incremental backup very easy - just upload the 
new SSTable files and you are done.



On 12/03/2021 13:06, David Tinker wrote:

Hi Guys

I need to backup my 3 node Cassandra cluster to a remote machine. Is 
there a tool like Barman (really nice streaming backup tool for 
Postgresql) for Cassandra? Or does everyone roll their own scripts 
using snapshots and so on?


The data is on all 3 nodes using about 900G of space on each.

It would be difficult for me to recover even a day of lost data. An 
hour might be ok.


Thanks
David



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: No node was available to execute query error

[Joe Obernberger]

Thank you Paul and Erick.  The keyspace is defined like this:
CREATE KEYSPACE doc WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '3'}  AND durable_writes = true;


Would that cause this?

The program that is having the problem selects data, calculates stuff, 
and inserts.  It works with smaller selects, but when the number of 
rows is in the millions, I start to get this error. Since it works with 
smaller sets, I don't believe it to be a network error.  All the nodes 
are definitely up as other processes are working OK, it's just this one 
program that fails.


The full stack trace:

Error: com.datastax.oss.driver.api.core.NoNodeAvailableException: No 
node was available to execute the query
com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was 
available to execute the query
        at 
com.datastax.oss.driver.api.core.NoNodeAvailableException.copy(NoNodeAvailableException.java:40)
        at 
com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
        at 
com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
        at 
com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
        at 
com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
        at 
com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
        at 
com.abc..fieldanalyzer.FTAProcess.udpateCassandraFTAMetrics(FTAProcess.java:275)
        at 
com.abc..fieldanalyzer.FTAProcess.storeResults(FTAProcess.java:216)
        at 
com.abc..fieldanalyzer.FTAProcess.startProcess(FTAProcess.java:199)

        at com.abc..fieldanalyzer.Main.main(Main.java:20)

FTAProcess like 275 is:

ResultSet rs = session.execute(getFieldCounts.bind().setString(0, 
rb.getSource()).setString(1, rb.getFieldName()));


-Joe

On 3/12/2021 8:30 AM, Paul Chandler wrote:

Hi Joe

This could also be caused by the replication factor of the keyspace, 
if you have NetworkTopologyStrategy and it doesn’t list a 
replication factor for the datacenter datacenter1 then you will get 
this error message too.Â


Paul

On 12 Mar 2021, at 13:07, Erick Ramirez > wrote:


Does it get returned by the driver every single time? The 
NoNodeAvailableExceptiongets thrown when (1) all nodes are down, or 
(2) all the contact points are invalid from the driver's perspective.


Is it possible there's no route/connectivity from your app server(s) 
to the 172.16.x.xnetwork? If you post the full error message + full 
stacktrace, it might provide clues. Cheers!



 
	Virus-free. www.avg.com 
 



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: No node was available to execute query error

[Paul Chandler]

Hi Joe

This could also be caused by the replication factor of the keyspace, if you 
have NetworkTopologyStrategy and it doesn’t list a replication factor for the 
datacenter datacenter1 then you will get this error message too. 

Paul

> On 12 Mar 2021, at 13:07, Erick Ramirez  wrote:
> 
> Does it get returned by the driver every single time? The 
> NoNodeAvailableException gets thrown when (1) all nodes are down, or (2) all 
> the contact points are invalid from the driver's perspective.
> 
> Is it possible there's no route/connectivity from your app server(s) to the 
> 172.16.x.x network? If you post the full error message + full stacktrace, it 
> might provide clues. Cheers!

Re: No node was available to execute query error

[Erick Ramirez]

Does it get returned by the driver every single time? The
NoNodeAvailableException gets thrown when (1) all nodes are down, or (2)
all the contact points are invalid from the driver's perspective.

Is it possible there's no route/connectivity from your app server(s) to the
172.16.x.x network? If you post the full error message + full stacktrace,
it might provide clues. Cheers!

Barman equivalent for Cassandra?

[David Tinker]

Hi Guys

I need to backup my 3 node Cassandra cluster to a remote machine. Is there
a tool like Barman (really nice streaming backup tool for Postgresql) for
Cassandra? Or does everyone roll their own scripts using snapshots and so
on?

The data is on all 3 nodes using about 900G of space on each.

It would be difficult for me to recover even a day of lost data. An hour
might be ok.

Thanks
David

No node was available to execute query error

[Joe Obernberger]

Hi All - I'm getting this error:

Error: com.datastax.oss.driver.api.core.NoNodeAvailableException: No 
node was available to execute the query
com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was 
available to execute the query
        at 
com.datastax.oss.driver.api.core.NoNodeAvailableException.copy(NoNodeAvailableException.java:40)
        at 
com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
        at 
com.datastax.oss.driver.internal.core.cql.CqlPrepareSyncProcessor.process(CqlPrepareSyncProcessor.java:59)
        at 
com.datastax.oss.driver.internal.core.cql.CqlPrepareSyncProcessor.process(CqlPrepareSyncProcessor.java:31)
        at 
com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
        at 
com.datastax.oss.driver.api.core.cql.SyncCqlSession.prepare(SyncCqlSession.java:224)


This is coming from the code:

ResultSet rs = session.execute(getFieldValueAndCount.bind().setString(0, 
rb.getSource()).setString(1, rb.getFieldName()));


The session connect (application.conf):

datastax-java-driver {
  basic.request.timeout = 60 seconds
  basic.request.consistency = ONE
  basic.contact-points = ["172.16.110.3:9042", "172.16.110.4:9042", 
"172.16.100.253:9042", "172.16.100.254:9042"]

  request.timeout = 60 seconds
  basic.load-balancing-policy {
        local-datacenter = datacenter1
  }
}

Any ideas?  nodetool status:

Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens       
Owns    Host 
ID                               Rack
UN  172.16.100.224  178 GiB    512          ?      
8ba646ac-2b33-49de-a220-ae9842f18806  rack1
UN  172.16.100.208  137.5 GiB  384          ?      
4e0ba42f-649b-425a-857a-34497eb3036e  rack1
UN  172.16.100.225  85.15 GiB  512          ?      
247f3d70-d13b-4d68-9a53-2ed58e01a63e  rack1
UN  172.16.110.3    62.33 GiB  768          ?      
0abea102-06d2-4309-af36-a3163e8f00d8  rack1
UN  172.16.110.4    170.11 GiB  512          ?      
2a5ae735-6304-4e99-924b-44d9d5ec86b7  rack1
UN  172.16.100.253  20.44 GiB  128          ?      
6b528b0b-d7f7-4378-bba8-1857802d4f18  rack1
UN  172.16.100.254  38.91 GiB  256          ?      
87d0cb48-a57d-460e-bd82-93e6e52e93ea  rack1


Note: Non-system keyspaces don't have the same replication settings, 
effective ownership information is meaningless


Thank you!

-Joe


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org