Re: Migrate from DSE (Datastax) to Apache Cassandra

[Ioannis Zafiropoulos]

Ok found a solution for this problem.
I deleted the system's keyspace directory and restarted COSS and it was
rebuilt.
rm -rf /var/lib/cassandra/data/system

A bit drastic but I'll test it also on a multi-node cluster.



On Thu, Aug 17, 2017 at 3:57 PM, Ioannis Zafiropoulos 
wrote:

> Thanks Felipe and Erick,
>
> Yes, your comment helped a lot, I was able to resolve that by:
> ALTER KEYSPACE dse_system WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor':'1'};
>
> Another problem I had was with CentOS release 6.7 (Final)
> I was getting glibc 2.14 not found.
> Based on this  I
> switched jna-4.4.0.jar with jna-4.2.2.jar and it worked.
>
> I just started COSS for the first time successfully, I am able to connect
> and work on the DB.
> It would be a perfect success if it was not for an exception that bugs me
> every time I start Cassandra:
>
> DEBUG [SSTableBatchOpen:1] 2017-08-17 14:36:50,477 SSTableReader.java:506
> - Opening 
> /cassandra/disk01/system/local-7ad54392bcdd35a684174e047860b377/mc-217-big
> (0.598KiB)
> DEBUG [SSTableBatchOpen:2] 2017-08-17 14:36:50,477 SSTableReader.java:506
> - Opening 
> /cassandra/disk01/system/local-7ad54392bcdd35a684174e047860b377/mc-155-big
> (0.139KiB)
> ERROR [SSTableBatchOpen:2] 2017-08-17 14:36:50,489
> DebuggableThreadPoolExecutor.java:239 - Error in ThreadPoolExecutor
> java.lang.RuntimeException: Unknown column server_id during deserialization
> at org.apache.cassandra.db.SerializationHeader$Component.
> toHeader(SerializationHeader.java:309) ~[apache-cassandra-3.11.0.jar:
> 3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:513)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_65]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_65]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at org.apache.cassandra.concurrent.NamedThreadFactory.
> lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.0.jar:3.11.0]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
>
> I looked at another DSE installation and system.local table has indeed a
> 'server_id' column.
> On my COSS testbed this column disappeared from the table as soon as I
> started for the first time COSS.
> I tried to sstablescrub, sstableupgrade but it didn't go away.
>
> I don't know if I should worry or how to fix it, any ideas?
>
>
>
> On Wed, Aug 16, 2017 at 1:38 PM, Felipe Esteves <
> felipe.este...@b2wdigital.com> wrote:
>
>> Ioannis,
>> As some people already said, there's one or two keyspaces that uses
>> EverywhereStrategy, dse_system is one of them, if I'm not wrong.
>> You  must remember to change them to a community strategy or it will fail.
>>
>>
>>
>>
>>
>

Re: Full table scan with cassandra

[Alex Kotelnikov]

So it is also terribly slow.

Does not work with materialized views, quick hack about that below and UDT,
this requires more time to fix.

So I used it to retrieve the only built-in type column, the key. To make
the task more time-consuming I exteneded the dataset a bit, to ~2.5M
records.

All of my multithreaded dumper, my spark job using connector and
cassandra-unload retrieve 2.5M records in ~1m20s. Increased concurrency
halves this time. But it works only with this trivial case fetching key
only, no actual payload.
Workload of fetching actual user data for this 2.5M is ~ 2m20s and it does
not go down with number of threads or spark cores.

I feel stupid. More than two minutes to transfer ~10Gb in a highly
distributed system.


index c570574..e7af28b 100644
--- a/src/main/java/com/datastax/loader/CqlDelimParser.java
+++ b/src/main/java/com/datastax/loader/CqlDelimParser.java
@@ -20,6 +20,7 @@ import com.datastax.driver.core.DataType;
 import com.datastax.driver.core.Row;
 import com.datastax.driver.core.Session;
 import com.datastax.driver.core.TableMetadata;
+import com.datastax.driver.core.AbstractTableMetadata;
 import com.datastax.driver.core.KeyspaceMetadata;
 import com.datastax.driver.core.exceptions.InvalidTypeException;
 import com.datastax.loader.parser.BigDecimalParser;
@@ -205,9 +206,10 @@ public class CqlDelimParser {
 System.err.println("Keyspace " + keyspace + " not found.");
 System.exit(-1);
 }
-TableMetadata tm = km.getTable(tablename);
+AbstractTableMetadata tm = km.getTable(tablename);
+if ( tm == null ) tm = km.getMaterializedView(tablename);
 if (null == tm) {
-System.err.println("Table " + tablename + " not found.");
+System.err.println("Table/view " + tablename + " not found.");
 System.exit(-1);
 }
 List inList = new ArrayList();
diff --git a/src/main/java/com/datastax/loader/CqlDelimUnload.java
b/src/main/java/com/datastax/loader/CqlDelimUnload.java
index 472e33b..f084ce8 100644
--- a/src/main/java/com/datastax/loader/CqlDelimUnload.java
+++ b/src/main/java/com/datastax/loader/CqlDelimUnload.java
@@ -56,6 +56,7 @@ import javax.net.ssl.KeyManagerFactory;
 import javax.net.ssl.SSLContext;
 import javax.net.ssl.TrustManagerFactory;

+import com.datastax.driver.core.AbstractTableMetadata;
 import com.datastax.driver.core.Cluster;
 import com.datastax.driver.core.Session;
 import com.datastax.driver.core.ColumnMetadata;
@@ -547,9 +548,12 @@ public class CqlDelimUnload {
 table = table.replaceAll("\"", "");
 else
 table = table.toLowerCase();
-
-List lcm = session.getCluster().getMetadata()
-.getKeyspace(keyspace).getTable(table).getPartitionKey();
+
+AbstractTableMetadata tm = session.getCluster().getMetadata()
+.getKeyspace(keyspace).getTable(table);
+if ( tm == null )
+  tm = session.getCluster().getMetada
ta().getKeyspace(keyspace).getMaterializedView(table);
+List lcm = tm.getPartitionKey();
 String partitionKey = lcm.get(0).getName();
 for (int i = 1; i < lcm.size(); i++) {
 partitionKey = partitionKey + "," + lcm.get(i).getName();


On 17 August 2017 at 21:11, Jeff Jirsa  wrote:

> Brian Hess has perhaps the best open source code example of the right way
> to do this:
>
> https://github.com/brianmhess/cassandra-loader/blob/master/
> src/main/java/com/datastax/loader/CqlDelimUnload.java
>
>
>
> On Thu, Aug 17, 2017 at 10:00 AM, Alex Kotelnikov <
> alex.kotelni...@diginetica.com> wrote:
>
>> yup, user_id is the primary key.
>>
>> First of all,can you share, how to "go to a node directly"?.
>>
>> Also such approach will retrieve all the data RF times, coordinator
>> should have enough metadata to avoid that.
>>
>> Should not requesting multiple coordinators provide certain concurrency?
>>
>> On 17 August 2017 at 19:54, Dor Laor  wrote:
>>
>>> On Thu, Aug 17, 2017 at 9:36 AM, Alex Kotelnikov <
>>> alex.kotelni...@diginetica.com> wrote:
>>>
 Dor,

 I believe, I tried it in many ways and the result is quite
 disappointing.
 I've run my scans on 3 different clusters, one of which was using on
 VMs and I was able to scale it up and down (3-5-7 VMs, 8 to 24 cores) to
 see, how this affects the performance.

 I also generated the flow from spark cluster ranging from 4 to 40
 parallel tasks as well as just multi-threaded client.

 The surprise is that trivial fetch of all records using token ranges
 takes pretty much the same time in all setups.

 The only beneficial thing I've learned is that it is much more
 efficient to create a MATERIALIZED VIEW than to filter (even using
 secondary index).

 Say, I have a typical dataset, around 3Gb of data, 1M records. And I
 have a trivial scan 

Re: Adding a new node with the double of disk space

[Jeff Jirsa]

If you really double the hardware in every way, it's PROBABLY reasonable to
double num_tokens. It won't be quite the same as doubling all-the-things,
because you still have a single JVM, and you'll still have to deal with GC
as you're now reading twice as much and generating twice as much garbage,
but you can probably adjust the tuning of the heap to compensate.



On Thu, Aug 17, 2017 at 1:00 PM, Kevin O'Connor 
wrote:

> Are you saying if a node had double the hardware capacity in every way it
> would be a bad idea to up num_tokens? I thought that was the whole idea of
> that setting though?
>
> On Thu, Aug 17, 2017 at 9:52 AM, Carlos Rolo  wrote:
>
>> No.
>>
>> If you would double all the hardware on that node vs the others would
>> still be a bad idea.
>> Keep the cluster uniform vnodes wise.
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
>> *linkedin.com/in/carlosjuzarterolo
>> *
>> Mobile: +351 918 918 100 <+351%20918%20918%20100>
>> www.pythian.com
>>
>> On Thu, Aug 17, 2017 at 5:47 PM, Cogumelos Maravilha <
>> cogumelosmaravi...@sapo.pt> wrote:
>>
>>> Hi all,
>>>
>>> I need to add a new node to my cluster but this time the new node will
>>> have the double of disk space comparing to the other nodes.
>>>
>>> I'm using the default vnodes (num_tokens: 256). To fully use the disk
>>> space in the new node I just have to configure num_tokens: 512?
>>>
>>> Thanks in advance.
>>>
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>>
>>
>> --
>>
>>
>>
>>
>

Re: Full table scan with cassandra

[Dmitry Saprykin]

Hi Alex,

How do you generate you subrange set for running queries?
It may happen that some of your ranges intersect data ownership range
borders (check it running 'nodetool describering [keyspace_name]')
Those range queries will be highly ineffective in that case and that could
explain your results.

Also you can think using LOCAL_ONE consistency for your full scans. You may
lose some consistency but will gain a log of performance improvements.

Kind regards,
Dmitry Saprykin

On Thu, Aug 17, 2017 at 12:36 PM, Alex Kotelnikov <
alex.kotelni...@diginetica.com> wrote:

> Dor,
>
> I believe, I tried it in many ways and the result is quite disappointing.
> I've run my scans on 3 different clusters, one of which was using on VMs
> and I was able to scale it up and down (3-5-7 VMs, 8 to 24 cores) to see,
> how this affects the performance.
>
> I also generated the flow from spark cluster ranging from 4 to 40 parallel
> tasks as well as just multi-threaded client.
>
> The surprise is that trivial fetch of all records using token ranges takes
> pretty much the same time in all setups.
>
> The only beneficial thing I've learned is that it is much more efficient
> to create a MATERIALIZED VIEW than to filter (even using secondary index).
>
> Say, I have a typical dataset, around 3Gb of data, 1M records. And I have
> a trivial scan practice:
>
> String.format("SELECT token(user_id), user_id, events FROM user_events
> WHERE token(user_id) >= %d ", start) + (end != null ? String.format(" AND
> token(user_id) < %d ", end) : "")
>
> I split all tokens into start-end ranges (except for last range, which
> only has start) and query ranges in multiple threads, up to 40.
>
> Whole process takes ~40s on 3 VMs cluster  2+2+4 cores, 16Gb RAM each 1
> virtual disk. And it takes ~30s on real hardware clusters
> 8servers*8cores*32Gb. Level of the concurrency does not matter pretty much
> at all. Util it is too high or too low.
> Size of tokens range matters, but here I see the rule "make it larger, but
> avoid cassandra timeouts".
> I also tried spark connector to validate that my test multithreaded app is
> not the bottleneck. It is not.
>
> I expected some kind of elasticity, I see none. Feels like I do something
> wrong...
>
>
>
> On 17 August 2017 at 00:19, Dor Laor  wrote:
>
>> Hi Alex,
>>
>> You probably didn't get the paralelism right. Serial scan has
>> a paralelism of one. If the paralelism isn't large enough, perf will be
>> slow.
>> If paralelism is too large, Cassandra and the disk will trash and have too
>> many context switches.
>>
>> So you need to find your cluster's sweet spot. We documented the procedure
>> to do it in this blog: http://www.scylladb.com/
>> 2017/02/13/efficient-full-table-scans-with-scylla-1-6/
>> and the results are here: http://www.scylladb.com/
>> 2017/03/28/parallel-efficient-full-table-scan-scylla/
>> The algorithm should translate to Cassandra but you'll have to use
>> different rules of the thumb.
>>
>> Best,
>> Dor
>>
>>
>> On Wed, Aug 16, 2017 at 9:50 AM, Alex Kotelnikov <
>> alex.kotelni...@diginetica.com> wrote:
>>
>>> Hey,
>>>
>>> we are trying Cassandra as an alternative for storage huge stream of
>>> data coming from our customers.
>>>
>>> Storing works quite fine, and I started to validate how retrieval does.
>>> We have two types of that: fetching specific records and bulk retrieval for
>>> general analysis.
>>> Fetching single record works like charm. But it is not so with bulk
>>> fetch.
>>>
>>> With a moderately small table of ~2 million records, ~10Gb raw data I
>>> observed very slow operation (using token(partition key) ranges). It takes
>>> minutes to perform full retrieval. We tried a couple of configurations
>>> using virtual machines, real hardware and overall looks like it is not
>>> possible to all table data in a reasonable time (by reasonable I mean that
>>> since we have 1Gbit network 10Gb can be transferred in a couple of minutes
>>> from one server to another and when we have 10+ cassandra servers and 10+
>>> spark executors total time should be even smaller).
>>>
>>> I tried datastax spark connector. Also I wrote a simple test case using
>>> datastax java driver and see how fetch of 10k records takes ~10s so I
>>> assume that "sequential" scan will take 200x more time, equals ~30 minutes.
>>>
>>> May be we are totally wrong trying to use Cassandra this way?
>>>
>>> --
>>>
>>> Best Regards,
>>>
>>>
>>> *Alexander Kotelnikov*
>>>
>>> *Team Lead*
>>>
>>> DIGINETICA
>>> Retail Technology Company
>>>
>>> m: +7.921.915.06.28 <+7%20921%20915-06-28>
>>>
>>> *www.diginetica.com *
>>>
>>
>>
>
>
> --
>
> Best Regards,
>
>
> *Alexander Kotelnikov*
>
> *Team Lead*
>
> DIGINETICA
> Retail Technology Company
>
> m: +7.921.915.06.28 <+7%20921%20915-06-28>
>
> *www.diginetica.com *
>

Re: Adding a new node with the double of disk space

[Kevin O'Connor]

Are you saying if a node had double the hardware capacity in every way it
would be a bad idea to up num_tokens? I thought that was the whole idea of
that setting though?

On Thu, Aug 17, 2017 at 9:52 AM, Carlos Rolo  wrote:

> No.
>
> If you would double all the hardware on that node vs the others would
> still be a bad idea.
> Keep the cluster uniform vnodes wise.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> *linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +351 918 918 100 <+351%20918%20918%20100>
> www.pythian.com
>
> On Thu, Aug 17, 2017 at 5:47 PM, Cogumelos Maravilha <
> cogumelosmaravi...@sapo.pt> wrote:
>
>> Hi all,
>>
>> I need to add a new node to my cluster but this time the new node will
>> have the double of disk space comparing to the other nodes.
>>
>> I'm using the default vnodes (num_tokens: 256). To fully use the disk
>> space in the new node I just have to configure num_tokens: 512?
>>
>> Thanks in advance.
>>
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>
> --
>
>
>
>

Re: Migrate from DSE (Datastax) to Apache Cassandra

[Ioannis Zafiropoulos]

Thanks Felipe and Erick,

Yes, your comment helped a lot, I was able to resolve that by:
ALTER KEYSPACE dse_system WITH replication = {'class': 'SimpleStrategy',
'replication_factor':'1'};

Another problem I had was with CentOS release 6.7 (Final)
I was getting glibc 2.14 not found.
Based on this  I
switched jna-4.4.0.jar with jna-4.2.2.jar and it worked.

I just started COSS for the first time successfully, I am able to connect
and work on the DB.
It would be a perfect success if it was not for an exception that bugs me
every time I start Cassandra:

DEBUG [SSTableBatchOpen:1] 2017-08-17 14:36:50,477 SSTableReader.java:506 -
Opening
/cassandra/disk01/system/local-7ad54392bcdd35a684174e047860b377/mc-217-big
(0.598KiB)
DEBUG [SSTableBatchOpen:2] 2017-08-17 14:36:50,477 SSTableReader.java:506 -
Opening
/cassandra/disk01/system/local-7ad54392bcdd35a684174e047860b377/mc-155-big
(0.139KiB)
ERROR [SSTableBatchOpen:2] 2017-08-17 14:36:50,489
DebuggableThreadPoolExecutor.java:239 - Error in ThreadPoolExecutor
java.lang.RuntimeException: Unknown column server_id during deserialization
at
org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:309)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:513)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.0.jar:3.11.0]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]

I looked at another DSE installation and system.local table has indeed a
'server_id' column.
On my COSS testbed this column disappeared from the table as soon as I
started for the first time COSS.
I tried to sstablescrub, sstableupgrade but it didn't go away.

I don't know if I should worry or how to fix it, any ideas?



On Wed, Aug 16, 2017 at 1:38 PM, Felipe Esteves <
felipe.este...@b2wdigital.com> wrote:

> Ioannis,
> As some people already said, there's one or two keyspaces that uses
> EverywhereStrategy, dse_system is one of them, if I'm not wrong.
> You  must remember to change them to a community strategy or it will fail.
>
>
>
>
>

Re: Full table scan with cassandra

[Jeff Jirsa]

Brian Hess has perhaps the best open source code example of the right way
to do this:

https://github.com/brianmhess/cassandra-loader/blob/master/src/main/java/com/datastax/loader/CqlDelimUnload.java



On Thu, Aug 17, 2017 at 10:00 AM, Alex Kotelnikov <
alex.kotelni...@diginetica.com> wrote:

> yup, user_id is the primary key.
>
> First of all,can you share, how to "go to a node directly"?.
>
> Also such approach will retrieve all the data RF times, coordinator should
> have enough metadata to avoid that.
>
> Should not requesting multiple coordinators provide certain concurrency?
>
> On 17 August 2017 at 19:54, Dor Laor  wrote:
>
>> On Thu, Aug 17, 2017 at 9:36 AM, Alex Kotelnikov <
>> alex.kotelni...@diginetica.com> wrote:
>>
>>> Dor,
>>>
>>> I believe, I tried it in many ways and the result is quite disappointing.
>>> I've run my scans on 3 different clusters, one of which was using on VMs
>>> and I was able to scale it up and down (3-5-7 VMs, 8 to 24 cores) to see,
>>> how this affects the performance.
>>>
>>> I also generated the flow from spark cluster ranging from 4 to 40
>>> parallel tasks as well as just multi-threaded client.
>>>
>>> The surprise is that trivial fetch of all records using token ranges
>>> takes pretty much the same time in all setups.
>>>
>>> The only beneficial thing I've learned is that it is much more efficient
>>> to create a MATERIALIZED VIEW than to filter (even using secondary index).
>>>
>>> Say, I have a typical dataset, around 3Gb of data, 1M records. And I
>>> have a trivial scan practice:
>>>
>>> String.format("SELECT token(user_id), user_id, events FROM user_events
>>> WHERE token(user_id) >= %d ", start) + (end != null ? String.format(" AND
>>> token(user_id) < %d ", end) : "")
>>>
>>
>> Is user_id the primary key? Looks like this query will just go to the
>> cluster and access a random coordinator each time.
>> C* doesn't save the subsequent token on the same node. It's hashed.
>> The idea of parallel cluster scan is to go directly to all nodes in
>> parallel and query them for the hashed keys they own.
>>
>>
>>> I split all tokens into start-end ranges (except for last range, which
>>> only has start) and query ranges in multiple threads, up to 40.
>>>
>>> Whole process takes ~40s on 3 VMs cluster  2+2+4 cores, 16Gb RAM each 1
>>> virtual disk. And it takes ~30s on real hardware clusters
>>> 8servers*8cores*32Gb. Level of the concurrency does not matter pretty much
>>> at all. Util it is too high or too low.
>>> Size of tokens range matters, but here I see the rule "make it larger,
>>> but avoid cassandra timeouts".
>>> I also tried spark connector to validate that my test multithreaded app
>>> is not the bottleneck. It is not.
>>>
>>> I expected some kind of elasticity, I see none. Feels like I do
>>> something wrong...
>>>
>>>
>>>
>>> On 17 August 2017 at 00:19, Dor Laor  wrote:
>>>
 Hi Alex,

 You probably didn't get the paralelism right. Serial scan has
 a paralelism of one. If the paralelism isn't large enough, perf will be
 slow.
 If paralelism is too large, Cassandra and the disk will trash and have
 too
 many context switches.

 So you need to find your cluster's sweet spot. We documented the
 procedure
 to do it in this blog: http://www.scylladb.com/
 2017/02/13/efficient-full-table-scans-with-scylla-1-6/
 and the results are here: http://www.scylladb.com/
 2017/03/28/parallel-efficient-full-table-scan-scylla/
 The algorithm should translate to Cassandra but you'll have to use
 different rules of the thumb.

 Best,
 Dor


 On Wed, Aug 16, 2017 at 9:50 AM, Alex Kotelnikov <
 alex.kotelni...@diginetica.com> wrote:

> Hey,
>
> we are trying Cassandra as an alternative for storage huge stream of
> data coming from our customers.
>
> Storing works quite fine, and I started to validate how retrieval
> does. We have two types of that: fetching specific records and bulk
> retrieval for general analysis.
> Fetching single record works like charm. But it is not so with bulk
> fetch.
>
> With a moderately small table of ~2 million records, ~10Gb raw data I
> observed very slow operation (using token(partition key) ranges). It takes
> minutes to perform full retrieval. We tried a couple of configurations
> using virtual machines, real hardware and overall looks like it is not
> possible to all table data in a reasonable time (by reasonable I mean that
> since we have 1Gbit network 10Gb can be transferred in a couple of minutes
> from one server to another and when we have 10+ cassandra servers and 10+
> spark executors total time should be even smaller).
>
> I tried datastax spark connector. Also I wrote a simple test case
> using datastax java driver and see how fetch of 10k records takes ~10s so 
> I
> assume that "sequential" scan 

Re: write time corrupted and not sure how

[Jeff Jirsa]

There are certainly cases where corruption has happened in cassandra (rare,
thankfully), but like I mentioned, I'm not aware of any that only corrupted
timestamps. It wouldn't surprise me to see a really broken clock, and it
wouldnt' surprise me to see bit flips on bad hardware (even hardware with
checksumming isn't guaranteed to catch all corruption issues, but if you're
using non-ECC RAM, of if you're doing TCP offloading on the NICs, bits
definitely flip from time to time).

Good luck.


On Thu, Aug 17, 2017 at 10:09 AM, Greg Saylor  wrote:

> Thanks for your help, I wrote a script to cycle through these early
> records and try to update them (some columns were missing, but could be
> gleaned from another db), then do the update, re-read, and if its not
> correct figure out the write time and re-issue the update with a timestamp
> + 1.  We’re exporting the data to a cluster so we can bring it into one
> with murmur3 partition, so hopefully this will address it.
>
> I don’t think this was a time shift thing, so far I’ve found 4 records in
> 2013 and one of them has a date like 3673-01-26 16:46:00 + - that would
> be quite a clock skew :)
>
> Thanks for the thoughtful reply.
>
> - Greg
>
> > On Aug 17, 2017, at 9:22 AM, Jeff Jirsa  wrote:
> >
> > It's a long, so you can't grab it with readInt - 8 bytes instead of 4
> >
> > You can delete it by issuing a delete with an explicit time stamp at
> least 1 higher the. The timestamp on the cell
> >
> > DELETE FROM table USING TIMESTAMP=? WHERE 
> >
> > https://cassandra.apache.org/doc/latest/cql/dml.html#delete
> >
> > This could happen if - in the past - one of your clients or servers had
> a very incorrect clock. I'm not aware of any bugs that corrupted timestamp
> anytime in the past 6-7 years of the database, but my memory isn't perfect.
> >
> >
> > --
> > Jeff Jirsa
> >
> >
> > On Aug 17, 2017, at 1:45 AM, Greg Saylor  wrote:
> >
> >> Hello,
> >>
> >> We have a Cassandra database that is about 5 years old and has gone
> through multiple upgrades.   Today I noticed a very odd thing (current
> timestamp would be around 1502957436214912):
> >>
> >>   cqlsh:siq_prod> select id,account_id,sweep_id from items where
> id=34681132;
> >>
> >>id   | account_id | sweep_id
> >>   --++--
> >>34681132 |  13896 |
> >>
> >> I then attempted to delete it:
> >>
> >>
> >> cqlsh:siq_prod> delete from items where id=34681132;
> >>
> >> But its still there, so I thought I’d look at the writteime of sweep_id:
> >>
> >> cqlsh:siq_prod> select id,account_id,sweep_id,writetime(sweep_id) from
> items where id=34681132;
> >>
> >> id   | account_id | sweep_id | writetime(sweep_id)
> >> --++--+-
> >> 34681132 |   null |  |1718969631988312
> >>
> >> That is Friday, June 21, 2024 11:33:51.988 AM
> >>
> >> Is there any way to get rid of this record or update the writetime?
> I’ve done a look around the database and there are many more examples of
> this.  There’s nothing we can think of that would have caused this, this
> record was inserted back in 2013 and there are other records within seconds
> of that one which are just fine.
> >>
> >> I suspect something must have gone awry during an upgrade or there was
> a subtle bug in the version of Cassandra we were running at the time.
> >>
> >> What started down this path was a tool an engineer was running that was
> written in NodeJS, apparently the cassandra driver for Node can’t parse
> this:
> >>
> >> { RangeError: Index out of range
> >>at checkOffset (buffer.js:821:11)
> >>at Buffer.readInt32BE (buffer.js:986:5)
> >>
> >> Most other drivers I’ve tested just return nil.
> >>
> >> Is there any way to get out of this situation?  I can’t delete it or
> update it.  This table has about 1 billion rows in it.
> >>
> >> Thank you,
> >>
> >> Greg Saylor
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: write time corrupted and not sure how

[Greg Saylor]

Thanks for your help, I wrote a script to cycle through these early records and 
try to update them (some columns were missing, but could be gleaned from 
another db), then do the update, re-read, and if its not correct figure out the 
write time and re-issue the update with a timestamp + 1.  We’re exporting the 
data to a cluster so we can bring it into one with murmur3 partition, so 
hopefully this will address it.

I don’t think this was a time shift thing, so far I’ve found 4 records in 2013 
and one of them has a date like 3673-01-26 16:46:00 + - that would be quite 
a clock skew :)

Thanks for the thoughtful reply.

- Greg

> On Aug 17, 2017, at 9:22 AM, Jeff Jirsa  wrote:
> 
> It's a long, so you can't grab it with readInt - 8 bytes instead of 4
> 
> You can delete it by issuing a delete with an explicit time stamp at least 1 
> higher the. The timestamp on the cell
> 
> DELETE FROM table USING TIMESTAMP=? WHERE  
> 
> https://cassandra.apache.org/doc/latest/cql/dml.html#delete
> 
> This could happen if - in the past - one of your clients or servers had a 
> very incorrect clock. I'm not aware of any bugs that corrupted timestamp 
> anytime in the past 6-7 years of the database, but my memory isn't perfect.
> 
> 
> -- 
> Jeff Jirsa
> 
> 
> On Aug 17, 2017, at 1:45 AM, Greg Saylor  wrote:
> 
>> Hello,
>> 
>> We have a Cassandra database that is about 5 years old and has gone through 
>> multiple upgrades.   Today I noticed a very odd thing (current timestamp 
>> would be around 1502957436214912):
>> 
>>   cqlsh:siq_prod> select id,account_id,sweep_id from items where id=34681132;
>> 
>>id   | account_id | sweep_id
>>   --++--
>>34681132 |  13896 | 
>> 
>> I then attempted to delete it:
>> 
>> 
>> cqlsh:siq_prod> delete from items where id=34681132;
>> 
>> But its still there, so I thought I’d look at the writteime of sweep_id:
>> 
>> cqlsh:siq_prod> select id,account_id,sweep_id,writetime(sweep_id) from items 
>> where id=34681132;
>> 
>> id   | account_id | sweep_id | writetime(sweep_id)
>> --++--+-
>> 34681132 |   null |  |1718969631988312
>> 
>> That is Friday, June 21, 2024 11:33:51.988 AM
>> 
>> Is there any way to get rid of this record or update the writetime?  I’ve 
>> done a look around the database and there are many more examples of this.  
>> There’s nothing we can think of that would have caused this, this record was 
>> inserted back in 2013 and there are other records within seconds of that one 
>> which are just fine.
>> 
>> I suspect something must have gone awry during an upgrade or there was a 
>> subtle bug in the version of Cassandra we were running at the time.
>> 
>> What started down this path was a tool an engineer was running that was 
>> written in NodeJS, apparently the cassandra driver for Node can’t parse this:
>> 
>> { RangeError: Index out of range
>>at checkOffset (buffer.js:821:11)
>>at Buffer.readInt32BE (buffer.js:986:5)
>> 
>> Most other drivers I’ve tested just return nil.
>> 
>> Is there any way to get out of this situation?  I can’t delete it or update 
>> it.  This table has about 1 billion rows in it.
>> 
>> Thank you,
>> 
>> Greg Saylor
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Full table scan with cassandra

[Alex Kotelnikov]

yup, user_id is the primary key.

First of all,can you share, how to "go to a node directly"?.

Also such approach will retrieve all the data RF times, coordinator should
have enough metadata to avoid that.

Should not requesting multiple coordinators provide certain concurrency?

On 17 August 2017 at 19:54, Dor Laor  wrote:

> On Thu, Aug 17, 2017 at 9:36 AM, Alex Kotelnikov <
> alex.kotelni...@diginetica.com> wrote:
>
>> Dor,
>>
>> I believe, I tried it in many ways and the result is quite disappointing.
>> I've run my scans on 3 different clusters, one of which was using on VMs
>> and I was able to scale it up and down (3-5-7 VMs, 8 to 24 cores) to see,
>> how this affects the performance.
>>
>> I also generated the flow from spark cluster ranging from 4 to 40
>> parallel tasks as well as just multi-threaded client.
>>
>> The surprise is that trivial fetch of all records using token ranges
>> takes pretty much the same time in all setups.
>>
>> The only beneficial thing I've learned is that it is much more efficient
>> to create a MATERIALIZED VIEW than to filter (even using secondary index).
>>
>> Say, I have a typical dataset, around 3Gb of data, 1M records. And I have
>> a trivial scan practice:
>>
>> String.format("SELECT token(user_id), user_id, events FROM user_events
>> WHERE token(user_id) >= %d ", start) + (end != null ? String.format(" AND
>> token(user_id) < %d ", end) : "")
>>
>
> Is user_id the primary key? Looks like this query will just go to the
> cluster and access a random coordinator each time.
> C* doesn't save the subsequent token on the same node. It's hashed.
> The idea of parallel cluster scan is to go directly to all nodes in
> parallel and query them for the hashed keys they own.
>
>
>> I split all tokens into start-end ranges (except for last range, which
>> only has start) and query ranges in multiple threads, up to 40.
>>
>> Whole process takes ~40s on 3 VMs cluster  2+2+4 cores, 16Gb RAM each 1
>> virtual disk. And it takes ~30s on real hardware clusters
>> 8servers*8cores*32Gb. Level of the concurrency does not matter pretty much
>> at all. Util it is too high or too low.
>> Size of tokens range matters, but here I see the rule "make it larger,
>> but avoid cassandra timeouts".
>> I also tried spark connector to validate that my test multithreaded app
>> is not the bottleneck. It is not.
>>
>> I expected some kind of elasticity, I see none. Feels like I do something
>> wrong...
>>
>>
>>
>> On 17 August 2017 at 00:19, Dor Laor  wrote:
>>
>>> Hi Alex,
>>>
>>> You probably didn't get the paralelism right. Serial scan has
>>> a paralelism of one. If the paralelism isn't large enough, perf will be
>>> slow.
>>> If paralelism is too large, Cassandra and the disk will trash and have
>>> too
>>> many context switches.
>>>
>>> So you need to find your cluster's sweet spot. We documented the
>>> procedure
>>> to do it in this blog: http://www.scylladb.com/
>>> 2017/02/13/efficient-full-table-scans-with-scylla-1-6/
>>> and the results are here: http://www.scylladb.com/
>>> 2017/03/28/parallel-efficient-full-table-scan-scylla/
>>> The algorithm should translate to Cassandra but you'll have to use
>>> different rules of the thumb.
>>>
>>> Best,
>>> Dor
>>>
>>>
>>> On Wed, Aug 16, 2017 at 9:50 AM, Alex Kotelnikov <
>>> alex.kotelni...@diginetica.com> wrote:
>>>
 Hey,

 we are trying Cassandra as an alternative for storage huge stream of
 data coming from our customers.

 Storing works quite fine, and I started to validate how retrieval does.
 We have two types of that: fetching specific records and bulk retrieval for
 general analysis.
 Fetching single record works like charm. But it is not so with bulk
 fetch.

 With a moderately small table of ~2 million records, ~10Gb raw data I
 observed very slow operation (using token(partition key) ranges). It takes
 minutes to perform full retrieval. We tried a couple of configurations
 using virtual machines, real hardware and overall looks like it is not
 possible to all table data in a reasonable time (by reasonable I mean that
 since we have 1Gbit network 10Gb can be transferred in a couple of minutes
 from one server to another and when we have 10+ cassandra servers and 10+
 spark executors total time should be even smaller).

 I tried datastax spark connector. Also I wrote a simple test case using
 datastax java driver and see how fetch of 10k records takes ~10s so I
 assume that "sequential" scan will take 200x more time, equals ~30 minutes.

 May be we are totally wrong trying to use Cassandra this way?

 --

 Best Regards,


 *Alexander Kotelnikov*

 *Team Lead*

 DIGINETICA
 Retail Technology Company

 m: +7.921.915.06.28 <+7%20921%20915-06-28>

 *www.diginetica.com *

>>>
>>>
>>
>>
>> --
>>
>> Best 

Re: Full table scan with cassandra

[Dor Laor]

On Thu, Aug 17, 2017 at 9:36 AM, Alex Kotelnikov <
alex.kotelni...@diginetica.com> wrote:

> Dor,
>
> I believe, I tried it in many ways and the result is quite disappointing.
> I've run my scans on 3 different clusters, one of which was using on VMs
> and I was able to scale it up and down (3-5-7 VMs, 8 to 24 cores) to see,
> how this affects the performance.
>
> I also generated the flow from spark cluster ranging from 4 to 40 parallel
> tasks as well as just multi-threaded client.
>
> The surprise is that trivial fetch of all records using token ranges takes
> pretty much the same time in all setups.
>
> The only beneficial thing I've learned is that it is much more efficient
> to create a MATERIALIZED VIEW than to filter (even using secondary index).
>
> Say, I have a typical dataset, around 3Gb of data, 1M records. And I have
> a trivial scan practice:
>
> String.format("SELECT token(user_id), user_id, events FROM user_events
> WHERE token(user_id) >= %d ", start) + (end != null ? String.format(" AND
> token(user_id) < %d ", end) : "")
>

Is user_id the primary key? Looks like this query will just go to the
cluster and access a random coordinator each time.
C* doesn't save the subsequent token on the same node. It's hashed.
The idea of parallel cluster scan is to go directly to all nodes in
parallel and query them for the hashed keys they own.


> I split all tokens into start-end ranges (except for last range, which
> only has start) and query ranges in multiple threads, up to 40.
>
> Whole process takes ~40s on 3 VMs cluster  2+2+4 cores, 16Gb RAM each 1
> virtual disk. And it takes ~30s on real hardware clusters
> 8servers*8cores*32Gb. Level of the concurrency does not matter pretty much
> at all. Util it is too high or too low.
> Size of tokens range matters, but here I see the rule "make it larger, but
> avoid cassandra timeouts".
> I also tried spark connector to validate that my test multithreaded app is
> not the bottleneck. It is not.
>
> I expected some kind of elasticity, I see none. Feels like I do something
> wrong...
>
>
>
> On 17 August 2017 at 00:19, Dor Laor  wrote:
>
>> Hi Alex,
>>
>> You probably didn't get the paralelism right. Serial scan has
>> a paralelism of one. If the paralelism isn't large enough, perf will be
>> slow.
>> If paralelism is too large, Cassandra and the disk will trash and have too
>> many context switches.
>>
>> So you need to find your cluster's sweet spot. We documented the procedure
>> to do it in this blog: http://www.scylladb.com/
>> 2017/02/13/efficient-full-table-scans-with-scylla-1-6/
>> and the results are here: http://www.scylladb.com/
>> 2017/03/28/parallel-efficient-full-table-scan-scylla/
>> The algorithm should translate to Cassandra but you'll have to use
>> different rules of the thumb.
>>
>> Best,
>> Dor
>>
>>
>> On Wed, Aug 16, 2017 at 9:50 AM, Alex Kotelnikov <
>> alex.kotelni...@diginetica.com> wrote:
>>
>>> Hey,
>>>
>>> we are trying Cassandra as an alternative for storage huge stream of
>>> data coming from our customers.
>>>
>>> Storing works quite fine, and I started to validate how retrieval does.
>>> We have two types of that: fetching specific records and bulk retrieval for
>>> general analysis.
>>> Fetching single record works like charm. But it is not so with bulk
>>> fetch.
>>>
>>> With a moderately small table of ~2 million records, ~10Gb raw data I
>>> observed very slow operation (using token(partition key) ranges). It takes
>>> minutes to perform full retrieval. We tried a couple of configurations
>>> using virtual machines, real hardware and overall looks like it is not
>>> possible to all table data in a reasonable time (by reasonable I mean that
>>> since we have 1Gbit network 10Gb can be transferred in a couple of minutes
>>> from one server to another and when we have 10+ cassandra servers and 10+
>>> spark executors total time should be even smaller).
>>>
>>> I tried datastax spark connector. Also I wrote a simple test case using
>>> datastax java driver and see how fetch of 10k records takes ~10s so I
>>> assume that "sequential" scan will take 200x more time, equals ~30 minutes.
>>>
>>> May be we are totally wrong trying to use Cassandra this way?
>>>
>>> --
>>>
>>> Best Regards,
>>>
>>>
>>> *Alexander Kotelnikov*
>>>
>>> *Team Lead*
>>>
>>> DIGINETICA
>>> Retail Technology Company
>>>
>>> m: +7.921.915.06.28 <+7%20921%20915-06-28>
>>>
>>> *www.diginetica.com *
>>>
>>
>>
>
>
> --
>
> Best Regards,
>
>
> *Alexander Kotelnikov*
>
> *Team Lead*
>
> DIGINETICA
> Retail Technology Company
>
> m: +7.921.915.06.28 <+7%20921%20915-06-28>
>
> *www.diginetica.com *
>

Re: Adding a new node with the double of disk space

[Carlos Rolo]

No.

If you would double all the hardware on that node vs the others would still
be a bad idea.
Keep the cluster uniform vnodes wise.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 918 918 100
www.pythian.com

On Thu, Aug 17, 2017 at 5:47 PM, Cogumelos Maravilha <
cogumelosmaravi...@sapo.pt> wrote:

> Hi all,
>
> I need to add a new node to my cluster but this time the new node will
> have the double of disk space comparing to the other nodes.
>
> I'm using the default vnodes (num_tokens: 256). To fully use the disk
> space in the new node I just have to configure num_tokens: 512?
>
> Thanks in advance.
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

-- 


--

Adding a new node with the double of disk space

[Cogumelos Maravilha]

Hi all,

I need to add a new node to my cluster but this time the new node will
have the double of disk space comparing to the other nodes.

I'm using the default vnodes (num_tokens: 256). To fully use the disk
space in the new node I just have to configure num_tokens: 512?

Thanks in advance.



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Full table scan with cassandra

[Alex Kotelnikov]

Dor,

I believe, I tried it in many ways and the result is quite disappointing.
I've run my scans on 3 different clusters, one of which was using on VMs
and I was able to scale it up and down (3-5-7 VMs, 8 to 24 cores) to see,
how this affects the performance.

I also generated the flow from spark cluster ranging from 4 to 40 parallel
tasks as well as just multi-threaded client.

The surprise is that trivial fetch of all records using token ranges takes
pretty much the same time in all setups.

The only beneficial thing I've learned is that it is much more efficient to
create a MATERIALIZED VIEW than to filter (even using secondary index).

Say, I have a typical dataset, around 3Gb of data, 1M records. And I have a
trivial scan practice:

String.format("SELECT token(user_id), user_id, events FROM user_events
WHERE token(user_id) >= %d ", start) + (end != null ? String.format(" AND
token(user_id) < %d ", end) : "")

I split all tokens into start-end ranges (except for last range, which only
has start) and query ranges in multiple threads, up to 40.

Whole process takes ~40s on 3 VMs cluster  2+2+4 cores, 16Gb RAM each 1
virtual disk. And it takes ~30s on real hardware clusters
8servers*8cores*32Gb. Level of the concurrency does not matter pretty much
at all. Util it is too high or too low.
Size of tokens range matters, but here I see the rule "make it larger, but
avoid cassandra timeouts".
I also tried spark connector to validate that my test multithreaded app is
not the bottleneck. It is not.

I expected some kind of elasticity, I see none. Feels like I do something
wrong...



On 17 August 2017 at 00:19, Dor Laor  wrote:

> Hi Alex,
>
> You probably didn't get the paralelism right. Serial scan has
> a paralelism of one. If the paralelism isn't large enough, perf will be
> slow.
> If paralelism is too large, Cassandra and the disk will trash and have too
> many context switches.
>
> So you need to find your cluster's sweet spot. We documented the procedure
> to do it in this blog: http://www.scylladb.com/
> 2017/02/13/efficient-full-table-scans-with-scylla-1-6/
> and the results are here: http://www.scylladb.com/
> 2017/03/28/parallel-efficient-full-table-scan-scylla/
> The algorithm should translate to Cassandra but you'll have to use
> different rules of the thumb.
>
> Best,
> Dor
>
>
> On Wed, Aug 16, 2017 at 9:50 AM, Alex Kotelnikov <
> alex.kotelni...@diginetica.com> wrote:
>
>> Hey,
>>
>> we are trying Cassandra as an alternative for storage huge stream of data
>> coming from our customers.
>>
>> Storing works quite fine, and I started to validate how retrieval does.
>> We have two types of that: fetching specific records and bulk retrieval for
>> general analysis.
>> Fetching single record works like charm. But it is not so with bulk fetch.
>>
>> With a moderately small table of ~2 million records, ~10Gb raw data I
>> observed very slow operation (using token(partition key) ranges). It takes
>> minutes to perform full retrieval. We tried a couple of configurations
>> using virtual machines, real hardware and overall looks like it is not
>> possible to all table data in a reasonable time (by reasonable I mean that
>> since we have 1Gbit network 10Gb can be transferred in a couple of minutes
>> from one server to another and when we have 10+ cassandra servers and 10+
>> spark executors total time should be even smaller).
>>
>> I tried datastax spark connector. Also I wrote a simple test case using
>> datastax java driver and see how fetch of 10k records takes ~10s so I
>> assume that "sequential" scan will take 200x more time, equals ~30 minutes.
>>
>> May be we are totally wrong trying to use Cassandra this way?
>>
>> --
>>
>> Best Regards,
>>
>>
>> *Alexander Kotelnikov*
>>
>> *Team Lead*
>>
>> DIGINETICA
>> Retail Technology Company
>>
>> m: +7.921.915.06.28 <+7%20921%20915-06-28>
>>
>> *www.diginetica.com *
>>
>
>


-- 

Best Regards,


*Alexander Kotelnikov*

*Team Lead*

DIGINETICA
Retail Technology Company

m: +7.921.915.06.28

*www.diginetica.com *

Re: write time corrupted and not sure how

[Jeff Jirsa]

It's a long, so you can't grab it with readInt - 8 bytes instead of 4

You can delete it by issuing a delete with an explicit time stamp at least 1 
higher the. The timestamp on the cell

DELETE FROM table USING TIMESTAMP=? WHERE  

https://cassandra.apache.org/doc/latest/cql/dml.html#delete

This could happen if - in the past - one of your clients or servers had a very 
incorrect clock. I'm not aware of any bugs that corrupted timestamp anytime in 
the past 6-7 years of the database, but my memory isn't perfect.


-- 
Jeff Jirsa


> On Aug 17, 2017, at 1:45 AM, Greg Saylor  wrote:
> 
> Hello,
> 
> We have a Cassandra database that is about 5 years old and has gone through 
> multiple upgrades.   Today I noticed a very odd thing (current timestamp 
> would be around 1502957436214912):
> 
>   cqlsh:siq_prod> select id,account_id,sweep_id from items where id=34681132;
> 
>id   | account_id | sweep_id
>   --++--
>34681132 |  13896 | 
> 
> I then attempted to delete it:
> 
> 
> cqlsh:siq_prod> delete from items where id=34681132;
> 
> But its still there, so I thought I’d look at the writteime of sweep_id:
> 
> cqlsh:siq_prod> select id,account_id,sweep_id,writetime(sweep_id) from items 
> where id=34681132;
> 
> id   | account_id | sweep_id | writetime(sweep_id)
> --++--+-
> 34681132 |   null |  |1718969631988312
> 
> That is Friday, June 21, 2024 11:33:51.988 AM
> 
> Is there any way to get rid of this record or update the writetime?  I’ve 
> done a look around the database and there are many more examples of this.  
> There’s nothing we can think of that would have caused this, this record was 
> inserted back in 2013 and there are other records within seconds of that one 
> which are just fine.
> 
> I suspect something must have gone awry during an upgrade or there was a 
> subtle bug in the version of Cassandra we were running at the time.
> 
> What started down this path was a tool an engineer was running that was 
> written in NodeJS, apparently the cassandra driver for Node can’t parse this:
> 
> { RangeError: Index out of range
>at checkOffset (buffer.js:821:11)
>at Buffer.readInt32BE (buffer.js:986:5)
> 
> Most other drivers I’ve tested just return nil.
> 
> Is there any way to get out of this situation?  I can’t delete it or update 
> it.  This table has about 1 billion rows in it.
> 
> Thank you,
> 
> Greg Saylor
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

write time corrupted and not sure how

[Greg Saylor]

Hello,

We have a Cassandra database that is about 5 years old and has gone through 
multiple upgrades.   Today I noticed a very odd thing (current timestamp would 
be around 1502957436214912):

   cqlsh:siq_prod> select id,account_id,sweep_id from items where id=34681132;

id   | account_id | sweep_id
   --++--
34681132 |  13896 | 

I then attempted to delete it:


cqlsh:siq_prod> delete from items where id=34681132;

But its still there, so I thought I’d look at the writteime of sweep_id:

cqlsh:siq_prod> select id,account_id,sweep_id,writetime(sweep_id) from items 
where id=34681132;

id   | account_id | sweep_id | writetime(sweep_id)
--++--+-
34681132 |   null |  |1718969631988312

That is Friday, June 21, 2024 11:33:51.988 AM

Is there any way to get rid of this record or update the writetime?  I’ve done 
a look around the database and there are many more examples of this.  There’s 
nothing we can think of that would have caused this, this record was inserted 
back in 2013 and there are other records within seconds of that one which are 
just fine.

I suspect something must have gone awry during an upgrade or there was a subtle 
bug in the version of Cassandra we were running at the time.

What started down this path was a tool an engineer was running that was written 
in NodeJS, apparently the cassandra driver for Node can’t parse this:

{ RangeError: Index out of range
at checkOffset (buffer.js:821:11)
at Buffer.readInt32BE (buffer.js:986:5)

Most other drivers I’ve tested just return nil.

Is there any way to get out of this situation?  I can’t delete it or update it. 
 This table has about 1 billion rows in it.

Thank you,

Greg Saylor
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org