Re: Upgrade from 3.11.5 to 4.1.x
You can check in your lower environment. On Fri, 11 Aug, 2023, 06:25 Surbhi Gupta, wrote: > Thanks, > > I am looking to to upgrade to 4.1.x . > Please advise. > > Thanks > Surbhi > > On Thu, Aug 10, 2023 at 5:39 PM MyWorld wrote: > >> Though it's recommended to upgrade to latest version of 3.11.x and then >> to ver 4 but even upgrading directly won't be a problem. Just check the >> release notes. >> >> However for production, I would recommend to go for 4.0.x latest stable >> version. >> >> Regards >> Ashish >> >> On Sat, 8 Jul, 2023, 05:44 Surbhi Gupta, >> wrote: >> >>> Hi, >>> >>> We have to upgrade from 3.11.5 to 4.1.x . >>> Can we do it in one go ? >>> Or do we have to go to an intermediate version first? >>> >>> Thanks >>> Surbhi >>> >>
Re: Upgrade from 3.11.5 to 4.1.x
Though it's recommended to upgrade to latest version of 3.11.x and then to ver 4 but even upgrading directly won't be a problem. Just check the release notes. However for production, I would recommend to go for 4.0.x latest stable version. Regards Ashish On Sat, 8 Jul, 2023, 05:44 Surbhi Gupta, wrote: > Hi, > > We have to upgrade from 3.11.5 to 4.1.x . > Can we do it in one go ? > Or do we have to go to an intermediate version first? > > Thanks > Surbhi >
Re: Materialized View inconsistency issue
Hi surbhi , There are 2 drawbacks associated with MV. 1. Inconsistent view 2. The lock it takes on the base table. This gets worse when you have huge number of clustering keys in a specific partition. It's better you re-design a seperate table and let your API do a parallel write on both. Regards, Ashish On Fri, 11 Aug, 2023, 02:03 Surbhi Gupta, wrote: > Thanks everyone. > > > On Wed, 9 Aug 2023 at 01:00, Regis Le Bretonnic > wrote: > > > > Hi Surbhi > > > > We do use cassandra materialized views even if not recommended. > > There are known issues you have to make with. Despite of them, we still > use VM. > > What we observe is : > > * there are inconsistency issues but few. Most of them are rows that > should not exist in the MV... > > * we made a spark script downloading the master table and the MV, and > comparing them and fixing data (as said previously we have very few errors > and we run it maybe once a year) > > > > * Things go very very very bad when you add or remove a node ! Limit > this operation if possible and do it knowing what can happen (we isolate > the ring/datacenter and fix data before putting it back to production. We > did this only once in the last 4 years). > > > > PS : all proposals avoiding MV failed for our project. Basically > managing a table like a MV (by deleting and inserting rows from code) is > worse and more corrupted than what MV does... > > The worse issue is adding and removing nodes. Maybe cassandra 4 improves > this point (not tested yet). > > > > Have fun... > > > > Le mar. 8 août 2023 à 22:36, Surbhi Gupta a > écrit : > >> > >> Hi, > >> > >> We get complaints about Materialized View inconsistency issues. > >> We are on 3.11.5 and on 3.11.5 Materialized Views were not production > ready. > >> We are ok to upgrade. > >> > >> On which version of cassandra MVs doesnt have inconsistency issues? > >> > >> Thanks > >> Surbhi >
Re: Cassandra p95 latencies
Hi, P95 should not be a problem if rightly designed. Levelled compaction strategy further reduces this, however it consume some resources. For read, caching is also helpful. Can you check your cpu iowait as it could be the reason for delay Regards, Ashish On Fri, 11 Aug, 2023, 04:58 Shaurya Gupta, wrote: > Hi community > > What is the expected P95 latency for Cassandra Read and Write queries > executed with Local_Quorum over a table with 3 replicas ? The queries are > done using the partition + clustering key and row size in bytes is not too > much, maybe 1-2 KB maximum. > Assuming CPU is not a crunch ? > > We observe those to be 40 ms P95 Reads and same for Writes. This looks > very high as compared to what we expected. We are using Cassandra 4.0. > > Any documentation / numbers will be helpful. > > Thanks > -- > Shaurya Gupta > > >
Re: Issue while node addition on cassandra 4.0.7
Bingo.. Yes API team was cross checkin their whitelisting to the db server at the same time and checking via telnet. Thanks Bowen On Thu, Jun 29, 2023 at 5:49 PM Bowen Song via user < user@cassandra.apache.org> wrote: > Talking about telnet, a closer look at the sequence 'FF F4 FF FD' makes > me thinking about telnet commands code. Based on RFC 854 > <https://www.rfc-editor.org/rfc/rfc854.html>, the sequence is IAC, > Interrupt Process, IAC, DO, which is basically the key sequence 'ctrl-c' > in telnet. > On 29/06/2023 12:42, Bowen Song wrote: > > Did anyone connecting to the servers' storage port via telnet, nc (netcat) > or something similar? 218762506 is 0x0D0A0D0A, which is two newlines. > > > On 29/06/2023 11:49, MyWorld wrote: > > When checked in the source nodes, we got similar errors. > > Forgot to mention, we also received below error message : > ERROR [Messaging-EventLoop-3-3] 2023-06-27 18:57:09,128 > InboundConnectionInitiator.java:360 - Failed to properly handshake with > peer /10.127.2.10:58490. Closing the channel. > io.netty.handler.codec.DecoderException: > org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read > 218762506, Expected -900387334 > > On Thu, Jun 29, 2023 at 2:57 PM Bowen Song via user < > user@cassandra.apache.org> wrote: > >> The expected value "-900387334" is the little endian decimal >> representation of the PROTOCOL_MAGIC value 0xCA552DFA defined in the >> net/Message.java >> <https://github.com/apache/cassandra/blob/c579faa488ec156a59ed8e15dd6db55759b9c942/src/java/org/apache/cassandra/net/Message.java#L393> >> file. >> >> The read value "-720899" converts to hex is 0xFFF4FFFD, that's not a >> valid TLS header which should start with 0x16, so I don't think has >> anything to do with the server encryption related options. It also does not >> look like a valid version number from pre-4.0 Cassandra, so we can rule >> that out too. Since it's nether a valid Cassandra 4.0+ magic, a TLS header >> nor a pre-4.0 version number, I have reason to believe the connection was >> not initiated by another Cassandra server for inter-node communication, but >> from another program. Can you follow the source IP and port number back to >> the originating host, and find out what is that program? or indeed it was >> one of the servers in the cluster, not something else, which could indicate >> a misconfiguration of the firewall rules. >> >> >> On 29/06/2023 01:26, MyWorld wrote: >> >> Hi all, >> We are currently using Apache cassandra 4.0.7 in our environment. While >> adding a new node in the existing 3-node DC, we found below error. >> This error is observed multiple times when the node was in the UJ (up and >> joining) state. >> >> Our current server-to-server internode encryption settings are default. >> server_encryption_options: >> internode_encryption: none >> enable_legacy_ssl_storage_port: false >> require_client_auth: false >> require_endpoint_verification: false >> >> Please help to debug the root cause of this error. >> Is it a point to worry about or is it just a Warning issue? >> Also, our API properties have received a few 5xx messages "Operation >> timed out. received only 2 responses" during this time(addition of new >> node), which we have not received when we were on the 3.11.x version. What >> could be the possible reason? >> However things are stable once the node comes to the UN state. >> >> ERROR [Messaging-EventLoop-3-10] 2023-06-27 18:37:14,931 >> InboundConnectionInitiator.java:360 - Failed to properly handshake with >> peer /x.x.x.x:35894. Closing the channel. >> io.netty.handler.codec.DecoderException: >> org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read -720899, >> Expected -900387334 >> at >> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) >> at >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) >> at >> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) >> at >> io.netty.channel.AbstractChannelHandlerContext.invo
Re: Issue while node addition on cassandra 4.0.7
When checked in the source nodes, we got similar errors. Forgot to mention, we also received below error message : ERROR [Messaging-EventLoop-3-3] 2023-06-27 18:57:09,128 InboundConnectionInitiator.java:360 - Failed to properly handshake with peer /10.127.2.10:58490. Closing the channel. io.netty.handler.codec.DecoderException: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read 218762506, Expected -900387334 On Thu, Jun 29, 2023 at 2:57 PM Bowen Song via user < user@cassandra.apache.org> wrote: > The expected value "-900387334" is the little endian decimal > representation of the PROTOCOL_MAGIC value 0xCA552DFA defined in the > net/Message.java > <https://github.com/apache/cassandra/blob/c579faa488ec156a59ed8e15dd6db55759b9c942/src/java/org/apache/cassandra/net/Message.java#L393> > file. > > The read value "-720899" converts to hex is 0xFFF4FFFD, that's not a > valid TLS header which should start with 0x16, so I don't think has > anything to do with the server encryption related options. It also does not > look like a valid version number from pre-4.0 Cassandra, so we can rule > that out too. Since it's nether a valid Cassandra 4.0+ magic, a TLS header > nor a pre-4.0 version number, I have reason to believe the connection was > not initiated by another Cassandra server for inter-node communication, but > from another program. Can you follow the source IP and port number back to > the originating host, and find out what is that program? or indeed it was > one of the servers in the cluster, not something else, which could indicate > a misconfiguration of the firewall rules. > > > On 29/06/2023 01:26, MyWorld wrote: > > Hi all, > We are currently using Apache cassandra 4.0.7 in our environment. While > adding a new node in the existing 3-node DC, we found below error. > This error is observed multiple times when the node was in the UJ (up and > joining) state. > > Our current server-to-server internode encryption settings are default. > server_encryption_options: > internode_encryption: none > enable_legacy_ssl_storage_port: false > require_client_auth: false > require_endpoint_verification: false > > Please help to debug the root cause of this error. > Is it a point to worry about or is it just a Warning issue? > Also, our API properties have received a few 5xx messages "Operation timed > out. received only 2 responses" during this time(addition of new node), > which we have not received when we were on the 3.11.x version. What > could be the possible reason? > However things are stable once the node comes to the UN state. > > ERROR [Messaging-EventLoop-3-10] 2023-06-27 18:37:14,931 > InboundConnectionInitiator.java:360 - Failed to properly handshake with > peer /x.x.x.x:35894. Closing the channel. > io.netty.handler.codec.DecoderException: > org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read -720899, > Expected -900387334 > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) > at > io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) > at > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: > Read -720899, Ex
Issue while node addition on cassandra 4.0.7
Hi all, We are currently using Apache cassandra 4.0.7 in our environment. While adding a new node in the existing 3-node DC, we found below error. This error is observed multiple times when the node was in the UJ (up and joining) state. Our current server-to-server internode encryption settings are default. server_encryption_options: internode_encryption: none enable_legacy_ssl_storage_port: false require_client_auth: false require_endpoint_verification: false Please help to debug the root cause of this error. Is it a point to worry about or is it just a Warning issue? Also, our API properties have received a few 5xx messages "Operation timed out. received only 2 responses" during this time(addition of new node), which we have not received when we were on the 3.11.x version. What could be the possible reason? However things are stable once the node comes to the UN state. ERROR [Messaging-EventLoop-3-10] 2023-06-27 18:37:14,931 InboundConnectionInitiator.java:360 - Failed to properly handshake with peer /x.x.x.x:35894. Closing the channel. io.netty.handler.codec.DecoderException: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read -720899, Expected -900387334 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read -720899, Expected -900387334 at org.apache.cassandra.net.Message.validateLegacyProtocolMagic(Message.java:340) at org.apache.cassandra.net.HandshakeProtocol$Initiate.maybeDecode(HandshakeProtocol.java:167) at org.apache.cassandra.net.InboundConnectionInitiator$Handler.initiate(InboundConnectionInitiator.java:242) at org.apache.cassandra.net.InboundConnectionInitiator$Handler.decode(InboundConnectionInitiator.java:235) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ... 15 common frames omitted Regards, Ashish
Issues during Install/Remove Cassandra ver 4.0.x
Hi all, We are facing one issue in installing cassandra-4.0.7. ### We started with* yum installation.* We setup repo "cassandra.repo" as below: [cassandra] name=Apache Cassandra baseurl=https://redhat.cassandra.apache.org/40x/noboolean/ gpgcheck=1 repo_gpgcheck=1 gpgkey=https://downloads.apache.org/cassandra/KEYS On doing "yum list cassandra", *it shows ver 4.0.8 but not 4.0.7*. Further using showduplicates "yum --showduplicates list cassandra", *still it shows ver 4.0.8 but not 4.0.7*. *How can we get earlier versions here ??* ###Next, we tried *using rpm,* sudo curl -OL https://redhat.cassandra.apache.org/40x/cassandra-4.0.7-1.noarch.rpm On running "sudo rpm -ivh cassandra-4.0.7-1.noarch.rpm", It gives below error, error: Failed dependencies: (jre-1.8.0 or jre-11) is needed by cassandra-4.0.7-1.noarch rpmlib(RichDependencies) <= 4.12.0-1 is needed by cassandra-4.0.7-1.noarch Then, i solve this by using "sudo rpm --nodeps -ivh cassandra-4.0.7-1.noarch.rpm" and version was installed successfully *Is skipping dependencies with --nodeps a right approach ??* ###Next, i tried to *uninstall the version* using "yum remove cassandra" It gives error: *Invalid version flag: or* Refer complete trace below: # yum remove cassandra Loaded plugins: fastestmirror Resolving Dependencies --> Running transaction check ---> Package cassandra.noarch 0:4.0.7-1 will be erased --> Finished Dependency Resolution Dependencies Resolved === Package Arch Version Repository Size === Removing: cassandra noarch 4.0.7-1 installed55 M Transaction Summary === Remove 1 Package Installed size: 55 M Is this ok [y/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Invalid version flag: or *How to solve this issue ??* Regards, Ashish Gupta
Re: Connection Latency with Cassandra 4.0.x
In addition to this, the established connection count to db is increasing
on every new API hit. Ideally existing connection should be reused.
Moreover even if we restart Cassandra the connection from the API end do
not get killed and remains the same.
On Thu, 12 Jan, 2023, 02:16 Elliott Sims, wrote:
> Consistently 200ms, during the back-and-forth negotiation rather than the
> handshake? That sounds suspiciously like Nagle interacting with Delayed
> ACK.
>
> On Wed, Jan 11, 2023 at 8:41 AM MyWorld wrote:
>
>> Hi all,
>> We are facing a connection latency of 200ms between API server and db
>> server during connection.
>> We are working with Apache cassandra 4.0.7 and open jdk ver 11.0.17. We
>> are using php on API side and connecting using php Cassandra driver (CPP
>> ver 2.7) with below string.
>> $cluster = Cassandra::cluster()
>> ->withContactPoints('x.x.x.x')
>> ->withPort(9042)
>> ->withCredentials("user", "pswd")
>> ->withPersistentSessions(true)
>> ->withDefaultConsistency(Cassandra::CONSISTENCY_LOCAL_QUORUM)
>> ->withDatacenterAwareRoundRobinLoadBalancingPolicy("dc1",0, false)
>> ->build();
>> $dbh = $cluster->connect('mykeyspace');
>>
>> We have earlier worked in 3.11.6 version and things were working fine.
>> This is the first time we have installed ver 4.0.x and started facing this
>> issue.No change have been done on driver side.
>> Just note both API and db server are on same location and ping time to db
>> server is less than 1ms.
>>
>> Unable to identify the root cause of this. Does anyone have any clue?
>>
>> Regards,
>> Ashish
>>
>>
> This email, including its contents and any attachment(s), may contain
> confidential and/or proprietary information and is solely for the review
> and use of the intended recipient(s). If you have received this email in
> error, please notify the sender and permanently delete this email, its
> content, and any attachment(s). Any disclosure, copying, or taking of any
> action in reliance on an email received in error is strictly prohibited.
>
Connection Latency with Cassandra 4.0.x
Hi all,
We are facing a connection latency of 200ms between API server and db
server during connection.
We are working with Apache cassandra 4.0.7 and open jdk ver 11.0.17. We are
using php on API side and connecting using php Cassandra driver (CPP ver
2.7) with below string.
$cluster = Cassandra::cluster()
->withContactPoints('x.x.x.x')
->withPort(9042)
->withCredentials("user", "pswd")
->withPersistentSessions(true)
->withDefaultConsistency(Cassandra::CONSISTENCY_LOCAL_QUORUM)
->withDatacenterAwareRoundRobinLoadBalancingPolicy("dc1",0, false)
->build();
$dbh = $cluster->connect('mykeyspace');
We have earlier worked in 3.11.6 version and things were working fine. This
is the first time we have installed ver 4.0.x and started facing this
issue.No change have been done on driver side.
Just note both API and db server are on same location and ping time to db
server is less than 1ms.
Unable to identify the root cause of this. Does anyone have any clue?
Regards,
Ashish
Re: Doubts on multiple filter design in cassandra
How many rows you are expecting within your partition? On Mon, 3 Oct, 2022, 21:56 Karthik K, wrote: > We have a table designed to retrieve products by name in ascending order. > OrganisationID and ProductType will be the compound partition key, whereas > the ProductName will be the clustering key. So, the primary key structure > is ((organisation_id, product_type), product_name) with clustering order > by(product_name asc). All have text as a datatype. > > We have 20-30 other attributes relevant to the product stored in other > different columns. Out of which some 5 attributes are significant. For > instance, those attributes can be description, colour, city, size and > date_of_manufacturing. All the above attributes are of text datatype except > for date_of_manufacturing which is a timestamp. > > Let's say a user wants to filter this product based on all these 5 > attributes. Can this be done using cassandra? Though we know that this can > be achieved using elastic search on top of cassandra, our constraint is to > use cassandra alone and achieve this. Storing data across many tables is > allowed. > > Note:At any instant, only 20 products can be listed in the page, which > means after applying all filters, we must display only 20 products. >
Re: Query around Data Modelling -2
Michiel, This is not in our use case. Since our data is not time series, there is no TTL in our case. Bowen, I think this is what is generally recommend to run a major compaction once in a week for better read performance. On Fri, Jul 1, 2022, 6:52 AM Michiel Saelen wrote: > Hi, > > We did do compaction job every week in the past to keep the disk space > used under control as we had mainly data in the table that needs to expire > with TTL and were also using levelled compaction. > > In our case we had different TTL’s in the same table and the partitions > were spread over multiple ssTables, as the partitions were never closing > and therefor kept on pushing changes we ended up with repair actions that > had to cover a lot of ssTables which is heavy on memory and CPU. > By changing the compaction strategy to TWCS > <https://cassandra.apache.org/doc/latest/cassandra/operating/compaction/twcs.html>, > splitting the table into different tables with their own TTL and adding a > part to the partition key (e.g. the day of the year) to close the > partitions, so they can be “marked” as repaired, we were able to get rid of > these heavy compaction actions. > > > > Not sure if you have the same use case, just wanted to share this info. > > > > Kind regards, > > Michiel > > > > <https://skyline.be/jobs/en> > > > > > > *Michiel Saelen *| Principal Solution Architect > > Email michiel.sae...@skyline.be > > > > Skyline Communications > > 39 Hong Kong Street #02-01 | Singapore 059678 > www.skyline.be | +65 6920 1145 <+6569201145> > > > > <https://skyline.be/> > > > > > > <https://teams.microsoft.com/l/chat/0/0?users=michiel.sae...@skyline.be> > > > <https://community.dataminer.services/?utm_source=signature_medium=email_campaign=icon> > > <https://www.linkedin.com/company/skyline-communications> > > <https://www.youtube.com/user/SkylineCommu> > > <https://www.facebook.com/SkylineCommunications/> > > <https://www.instagram.com/skyline.dataminer/> > > > <https://skyline.be/skyline/awards?utm_source=signature_medium=email_campaign=icon> > > > > > > > > *From:* Bowen Song > *Sent:* Friday, July 1, 2022 08:48 > *To:* user@cassandra.apache.org > *Subject:* Re: Query around Data Modelling -2 > > > > This message was sent from outside the company. Please do not click links > or open attachments unless you recognise the source of this email and know > the content is safe. > > > > And why do you do that? > > On 30/06/2022 16:35, MyWorld wrote: > > We run major compaction once in a week > > > > On Thu, Jun 30, 2022, 8:14 PM Bowen Song wrote: > > I have noticed this "running a weekly repair and compaction job". > > What do you mean weekly compaction job? Have you disabled the > auto-compaction on the table and is relying on weekly scheduled > compactions? Or running weekly major compactions? Neither of these sounds > right. > > On 30/06/2022 15:03, MyWorld wrote: > > Hi all, > > > > Another query around data Modelling. > > > > We have a existing table with below structure: > > Table(PK,CK, col1,col2, col3, col4,col5) > > > > Now each Pk here have 1k - 10k Clustering keys. Each PK has size from 10MB > to 80MB. We have overall 100+ millions partitions. Also we have set > levelled compactions in place so as to get better read response time. > > > > We are currently on 3.11.x version of Cassandra. On running a weekly > repair and compaction job, this model because of levelled compaction > (occupied till Level 3) consume heavy cpu resource and impact db > performance. > > > > Now what if we divide this table in 10 with each table containing 1/10 > partitions. So now each table will be limited to levelled compaction upto > level-2. I think this would ease down read as well as compaction task. > > > > What is your opinion on this? > > Even if we upgrade to ver 4.0, is the second model ok? > > > >
Re: Query around Data Modelling -2
We run major compaction once in a week On Thu, Jun 30, 2022, 8:14 PM Bowen Song wrote: > I have noticed this "running a weekly repair and compaction job". > > What do you mean weekly compaction job? Have you disabled the > auto-compaction on the table and is relying on weekly scheduled > compactions? Or running weekly major compactions? Neither of these sounds > right. > On 30/06/2022 15:03, MyWorld wrote: > > Hi all, > > Another query around data Modelling. > > We have a existing table with below structure: > Table(PK,CK, col1,col2, col3, col4,col5) > > Now each Pk here have 1k - 10k Clustering keys. Each PK has size from 10MB > to 80MB. We have overall 100+ millions partitions. Also we have set > levelled compactions in place so as to get better read response time. > > We are currently on 3.11.x version of Cassandra. On running a weekly > repair and compaction job, this model because of levelled compaction > (occupied till Level 3) consume heavy cpu resource and impact db > performance. > > Now what if we divide this table in 10 with each table containing 1/10 > partitions. So now each table will be limited to levelled compaction upto > level-2. I think this would ease down read as well as compaction task. > > What is your opinion on this? > Even if we upgrade to ver 4.0, is the second model ok? > >
Re: Query around Data Modelling -2
Hi Jeff, We are running repair with -pr option. You are right it would have no or very minimal impact on read (considering the fact now data has to be read from 2 levels instead of 3). But my guess there is no negative impact of this model2. On Thu, Jun 30, 2022, 7:41 PM Jeff Jirsa wrote: > How are you running repair? -pr? Or -st/-et? > > 4.0 gives you real incremental repair which helps. Splitting the table > won’t make reads faster. It will increase the potential parallelization of > compaction. > > On Jun 30, 2022, at 7:04 AM, MyWorld wrote: > > > Hi all, > > Another query around data Modelling. > > We have a existing table with below structure: > Table(PK,CK, col1,col2, col3, col4,col5) > > Now each Pk here have 1k - 10k Clustering keys. Each PK has size from 10MB > to 80MB. We have overall 100+ millions partitions. Also we have set > levelled compactions in place so as to get better read response time. > > We are currently on 3.11.x version of Cassandra. On running a weekly > repair and compaction job, this model because of levelled compaction > (occupied till Level 3) consume heavy cpu resource and impact db > performance. > > Now what if we divide this table in 10 with each table containing 1/10 > partitions. So now each table will be limited to levelled compaction upto > level-2. I think this would ease down read as well as compaction task. > > What is your opinion on this? > Even if we upgrade to ver 4.0, is the second model ok? > >
Query around Data Modelling -2
Hi all, Another query around data Modelling. We have a existing table with below structure: Table(PK,CK, col1,col2, col3, col4,col5) Now each Pk here have 1k - 10k Clustering keys. Each PK has size from 10MB to 80MB. We have overall 100+ millions partitions. Also we have set levelled compactions in place so as to get better read response time. We are currently on 3.11.x version of Cassandra. On running a weekly repair and compaction job, this model because of levelled compaction (occupied till Level 3) consume heavy cpu resource and impact db performance. Now what if we divide this table in 10 with each table containing 1/10 partitions. So now each table will be limited to levelled compaction upto level-2. I think this would ease down read as well as compaction task. What is your opinion on this? Even if we upgrade to ver 4.0, is the second model ok?
Re: Query around Data Modelling
Thanks a lot Jeff, Michiel and Manish for your replies. Really helpful. On Thu, Jun 23, 2022, 9:50 AM Jeff Jirsa wrote: > This is assuming each row is like … I dunno 10-1000 bytes. If you’re > storing like a huge 1mb blob use two tables for sure. > > On Jun 22, 2022, at 9:06 PM, Jeff Jirsa wrote: > > > > Ok so here’s how I would think about this > > The writes don’t matter. (There’s a tiny tiny bit of nuance in one table > where you can contend adding to the memtable but the best cassandra > engineers on earth probably won’t notice that unless you have really super > hot partitions, so ignore the write path). > > The reads are where it changes > > In both models/cases, you’ll use the partition index to seek to where the > partition starts. > > In model 2 table 1 if you use ck+col1+… the read will load the column > index and use that to jump to within 64kb of the col1 value you specify > > In model 2 table 2, if you use ck+col3+…, same thing - column index can > jump to within 64k > > What you give up in model one is the granularity of that jump. If you use > model 1 and col3 instead of col1, the read will have to scan the partition. > In your case, with 80 rows, that may still be within that 64kb block - you > may not get more granular than that anyway. And even if it’s slightly > larger, you’re probably going to be compressing 64k chunks - maybe you have > to decompress one extra chunk on read if your 1000 rows goes past 64k, but > you likely won’t actually notice. You’re technically asking the server to > read and skip data it doesn’t need to return - it’s not really the most > efficient, but at that partition size it’s noise. You could also just > return all 80-100 rows, let the server do slightly less work and filter > client side - also valid, probably slightly worse than the server side > filter. > > Having one table instead of two, though, probably saves you a ton of disk > space ($€£), and the lower disk space may also mean that data stays in page > cache, so the extra read may not even go to disk anyway. > > So with your actual data shape, I imagine you won’t really notice the > nominal inefficiency of the first model, and I’d be inclined to do that > until you demonstrate it won’t work (I bet it works fine for a long long > time). > > On Jun 22, 2022, at 7:11 PM, MyWorld wrote: > > > Hi Jeff, > Let me know how no of rows have an impact here. > May be today I have 80-100 rows per partition. But what if I started > storing 2-4k rows per partition. However total partition size is still > under 100 MB > > On Thu, Jun 23, 2022, 7:18 AM Jeff Jirsa wrote: > >> How many rows per partition in each model? >> >> >> > On Jun 22, 2022, at 6:38 PM, MyWorld wrote: >> > >> > >> > Hi all, >> > >> > Just a small query around data Modelling. >> > Suppose we have to design the data model for 2 different use cases >> which will query the data on same set of (partion+clustering key). So >> should we maintain a seperate table for each or a single table. >> > >> > Model1 - Combined table >> > Table(Pk,CK, col1,col2, col3, col4,col5) >> > >> > Model2 - Seperate tables >> > Table1(Pk,CK,col1,col2,col3) >> > Table2(Pk,CK,col3,col4,col45) >> > >> > So here partion and clustering keys are same. Also note column col3 is >> required in both use cases. >> > >> > As per my thought in Model2, partition size would be less. There would >> be less sstables and when I use level compaction, it could be easily >> maintained. So should be better read performance. >> > >> > Please help me to highlight the drawback and advantage of each data >> model. Here we have a mix kind of workload (read/write) >> >
Re: Query around Data Modelling
Hi Jeff, Let me know how no of rows have an impact here. May be today I have 80-100 rows per partition. But what if I started storing 2-4k rows per partition. However total partition size is still under 100 MB On Thu, Jun 23, 2022, 7:18 AM Jeff Jirsa wrote: > How many rows per partition in each model? > > > > On Jun 22, 2022, at 6:38 PM, MyWorld wrote: > > > > > > Hi all, > > > > Just a small query around data Modelling. > > Suppose we have to design the data model for 2 different use cases which > will query the data on same set of (partion+clustering key). So should we > maintain a seperate table for each or a single table. > > > > Model1 - Combined table > > Table(Pk,CK, col1,col2, col3, col4,col5) > > > > Model2 - Seperate tables > > Table1(Pk,CK,col1,col2,col3) > > Table2(Pk,CK,col3,col4,col45) > > > > So here partion and clustering keys are same. Also note column col3 is > required in both use cases. > > > > As per my thought in Model2, partition size would be less. There would > be less sstables and when I use level compaction, it could be easily > maintained. So should be better read performance. > > > > Please help me to highlight the drawback and advantage of each data > model. Here we have a mix kind of workload (read/write) >
Query around Data Modelling
Hi all, Just a small query around data Modelling. Suppose we have to design the data model for 2 different use cases which will query the data on same set of (partion+clustering key). So should we maintain a seperate table for each or a single table. Model1 - Combined table Table(Pk,CK, col1,col2, col3, col4,col5) Model2 - Seperate tables Table1(Pk,CK,col1,col2,col3) Table2(Pk,CK,col3,col4,col45) So here partion and clustering keys are same. Also note column col3 is required in both use cases. As per my thought in Model2, partition size would be less. There would be less sstables and when I use level compaction, it could be easily maintained. So should be better read performance. Please help me to highlight the drawback and advantage of each data model. Here we have a mix kind of workload (read/write)
sstablepartitions in apache cassandra
Hi all, I want to check all partitions greater than a certain threshold in my existing apache cassandra column family. Is there any alternative for datastax sstable tool 'sstablepartitions' in open source apache cassandra? Regards, Ashish Gupta
Removing row level tombstones
Hi all,
I have a table (with default gc_grace_seconds) where we have some row
tombstones created. Currently there are 2 sstables for this.
When we check the sstablemetadata, we found Estimated droppable tombstones:
0.689(for sstable1) and 1.094(for sstable2)
After executing nodetool compact on this table, we found that row
tombstones still exist.
Even after executing nodetool garbagecollect and compact one after another,
we were not able to delete these tombstones.
Please suggest how to remove these.
*Sample of sstabledump (for sstable 1)*
[
{
"partition" : {
"key" : [ "114168" ],
"position" : 733
},
"rows" : [
{
"type" : "row",
"position" : 755,
"clustering" : [ "Disposable Catheter" ],
"deletion_info" : { "marked_deleted" : "2020-10-12T08:57:57.015Z",
"local_delete_time" : "2020-10-12T08:57:57Z" },
"cells" : [ ]
},
{
"type" : "row",
"position" : 791,
"clustering" : [ "Disposable Catheter Akash" ],
"deletion_info" : { "marked_deleted" : "2020-10-12T09:01:28.927Z",
"local_delete_time" : "2020-10-12T09:01:28Z" },
"cells" : [ ]
},
{
"type" : "row",
"position" : 869,
"clustering" : [ "Single Use Catheter test" ],
"liveness_info" : { "tstamp" : "2020-10-12T09:03:00.223Z" },
"cells" : [
{ "name" : "my_id", "value" : 1 }
]
}
]
}
]
*Sample of sstabledump (for sstable 2)*
[
{
"partition" : {
"key" : [ "114168" ],
"position" : 544360
},
"rows" : [
{
"type" : "row",
"position" : 544382,
"clustering" : [ "Disposable Catheter" ],
"deletion_info" : { "marked_deleted" : "2020-10-12T08:57:57.015Z",
"local_delete_time" : "2020-10-12T08:57:57Z" },
"cells" : [ ]
},
{
"type" : "row",
"position" : 544418,
"clustering" : [ "Disposable Catheter Akash" ],
"deletion_info" : { "marked_deleted" : "2020-10-12T09:01:28.927Z",
"local_delete_time" : "2020-10-12T09:01:28Z" },
"cells" : [ ]
},
{
"type" : "row",
"position" : 544496,
"clustering" : [ "Single Use Catheter test" ],
"liveness_info" : { "tstamp" : "2020-10-12T09:03:00.223Z" },
"cells" : [
{ "name" : "my_id", "value" : 1 }
]
},
{
"type" : "row",
"position" : 544551,
"clustering" : [ "Catheter" ],
"liveness_info" : { "tstamp" : "2020-05-26T08:07:38.153026Z" },
"cells" : [
{ "name" : "my_id", "value" : 1 }
]
}
]
}
]
Regards,
Ashish
Re: Need urgent help in cassandra modelling
Anyone have any clue? On Wed, Mar 9, 2022 at 7:01 PM MyWorld wrote: > Hi all, > Some problems with the display. Resending my query- > > I am modelling a table for a shopping site where we store products for > customers and their data in json. Max prods for a customer is 10k. > > We initially designed this table with the architecture below: > cust_prods(cust_id bigint PK, prod_id bigint CK, prod_data text). > cust_id is partition key, prod_id is clustering key and combination of > these is unique. > > Now, we have a requirement to store one ordering column "prod_order > (bigint)" which is pre-calculated from service end after complex > calculation. > Further we now want to want to display products in pagination (100 per > page in order of prod_order) > > Please suggest the new architecture for the table "cust_prods" so that : > 1. We could be still able to update a single prod info based on cust_id + > prod_id > 2. We could store data in order of "prod_order" so that fetching in > limit(pagination) becomes easy. > > Note : prod_order could change for a product frequently > > Regards, > Ashish > > On Wed, Mar 9, 2022 at 6:55 PM MyWorld wrote: > >> Hi all, >> >> I am modelling a table for a shopping site where we store products for >> customers and their data in json. Max prods for a customer is 10k. >> >> >>We initially designed this table with the architecture below: >> cust_prods(cust_id bigint PK, prod_id bigint CK, prod_data text). >> cust_id is partition key, prod_id is clustering key and combination of >> these is unique. >> >> >>Now, we have a requirement to store one ordering column "prod_order >> (bigint)" which is pre-calculated from service end after complex >> calculation. >> Further we now want to want to display products in pagination (100 per >> page in order of prod_order) >> >> Please suggest the new architecture for the table "cust_prods" so that : >> 1. We could be still able to update a single prod info based on cust_id + >> prod_id >> 2. We could store data in order of "prod_order" so that fetching in >> limit(pagination) becomes easy. >> >> Note : prod_order could change for a product frequently >> >> Regards, >> Ashish >> >
Re: Need urgent help in cassandra modelling
Hi all, Some problems with the display. Resending my query- I am modelling a table for a shopping site where we store products for customers and their data in json. Max prods for a customer is 10k. We initially designed this table with the architecture below: cust_prods(cust_id bigint PK, prod_id bigint CK, prod_data text). cust_id is partition key, prod_id is clustering key and combination of these is unique. Now, we have a requirement to store one ordering column "prod_order (bigint)" which is pre-calculated from service end after complex calculation. Further we now want to want to display products in pagination (100 per page in order of prod_order) Please suggest the new architecture for the table "cust_prods" so that : 1. We could be still able to update a single prod info based on cust_id + prod_id 2. We could store data in order of "prod_order" so that fetching in limit(pagination) becomes easy. Note : prod_order could change for a product frequently Regards, Ashish On Wed, Mar 9, 2022 at 6:55 PM MyWorld wrote: > Hi all, > > I am modelling a table for a shopping site where we store products for > customers and their data in json. Max prods for a customer is 10k. > > >>We initially designed this table with the architecture below: > cust_prods(cust_id bigint PK, prod_id bigint CK, prod_data text). > cust_id is partition key, prod_id is clustering key and combination of > these is unique. > > >>Now, we have a requirement to store one ordering column "prod_order > (bigint)" which is pre-calculated from service end after complex > calculation. > Further we now want to want to display products in pagination (100 per > page in order of prod_order) > > Please suggest the new architecture for the table "cust_prods" so that : > 1. We could be still able to update a single prod info based on cust_id + > prod_id > 2. We could store data in order of "prod_order" so that fetching in > limit(pagination) becomes easy. > > Note : prod_order could change for a product frequently > > Regards, > Ashish >
Need urgent help in cassandra modelling
Hi all, I am modelling a table for a shopping site where we store products for customers and their data in json. Max prods for a customer is 10k. >>We initially designed this table with the architecture below: cust_prods(cust_id bigint PK, prod_id bigint CK, prod_data text). cust_id is partition key, prod_id is clustering key and combination of these is unique. >>Now, we have a requirement to store one ordering column "prod_order (bigint)" which is pre-calculated from service end after complex calculation. Further we now want to want to display products in pagination (100 per page in order of prod_order) Please suggest the new architecture for the table "cust_prods" so that : 1. We could be still able to update a single prod info based on cust_id + prod_id 2. We could store data in order of "prod_order" so that fetching in limit(pagination) becomes easy. Note : prod_order could change for a product frequently Regards, Ashish
Re: Streaming failure Issue
Hi Jim, It's 600 Megabits not MegaBytes. So near around 600/8=75MBps. Also, streaming is happening from any 3 nodes at a time. However we have tried with default Streaming throughput which is 200 Megabits per sec (25MBps), but still the same issue. Heap we have setup to 8gb on GCP and seems pretty much normal. On Thu, Oct 7, 2021, 4:05 AM Jim Shaw wrote: > I met similar issue before. What I did was: reduce Heap size for > rebuild, reduce streamthroughput. > But it depends on version, and your env., may not your case, just hope it > helpful. > > ps -ef | grep , you will see a new java process for rebuild, see what > memory size used, if use default, it may be use too much, just export > MAX_HEAP_SIZE before nodetool rebuild, it will limit heap size. > > streamthroughput=600MB/s, if you look by nodetool, or OS level file, or > the log, you will it pull files from all nodes --- that is 5 in your > case. so it will be 3 GB/s , on-premise side may not handle it due to > firewall setting. > > Regards, > Jim > > On Tue, Oct 5, 2021 at 8:43 AM MyWorld wrote: > >> Logged "nodetool failuredetector" every 5sec. Doesn't seems to be an >> issue for phi_convict_threshold value >> >> On Tue, Oct 5, 2021 at 4:35 PM Surbhi Gupta >> wrote: >> >>> Hi , >>> >>> Try to adjust phi_convict_threshold and see if that helps. >>> When we did migration from on prim to AWS, this was one of the factor to >>> consider. >>> >>> Thanks >>> >>> >>> On Tue, Oct 5, 2021 at 4:00 AM MyWorld wrote: >>> >>>> Hi all, >>>> >>>> Need urgent help. >>>> We have one Physical Data Center of 5 nodes with 1 TB data on each >>>> (Location: Dallas). Currently we are using Cassandra ver 3.0.9. Now we are >>>> Adding one more Data Center of 5 nodes(Location GCP-US) and have joined it >>>> to the existing one. >>>> >>>> While running nodetool rebuild command, we are getting following error : >>>> On GCP node (where we ran rebuild command) : >>>> >>>>> ERROR [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,246 >>>>> StreamSession.java:639 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] >>>>> Remote peer 192.x.x.x failed stream session. >>>>> INFO [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,266 >>>>> StreamResultFuture.java:183 - [Stream >>>>> #66646d30-25a2-11ec-903b-774f88efe725] Session with /192.x.x.x is complete >>>> >>>> >>>> On DL source node : >>>> >>>>> INFO [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,785 >>>>> StreamResultFuture.java:183 - [Stream >>>>> #66646d30-25a2-11ec-903b-774f88efe725] Session with /34.x.x.x is complete >>>>> ERROR [STREAM-OUT-/34.x.x.x] 2021-10-05 15:55:53,785 >>>>> StreamSession.java:534 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] >>>>> Streaming error occurred >>>>> java.lang.RuntimeException: Transfer of file >>>>> /var/lib/cassandra/data/clickstream/glusr_usr_paid_url_mv-3c49c392b35511e9bd0a8f42dfb09617/mc-45676-big-Data.db >>>>> already completed or aborted (perhaps session failed?). >>>>> at >>>>> org.apache.cassandra.streaming.messages.OutgoingFileMessage.startTransfer(OutgoingFileMessage.java:120) >>>>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>>>> at >>>>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50) >>>>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>>>> at >>>>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42) >>>>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>>>> at >>>>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:48) >>>>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>>>> at >>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:387) >>>>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>>>> at >>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:367) >>>>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>>>> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192] >>>>> WARN [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,786 >>>>> StreamResultFuture.java:210 - [Stream >>>>> #66646d30-25a2-11ec-903b-774f88efe725] Stream failed >>>> >>>> >>>> Before starting this rebuild, we have made the following changes: >>>> 1. Set setstreamthroughput to 600 Mb/sec >>>> 2. Set setinterdcstreamthroughput to 600 Mb/sec >>>> 3. streaming_socket_timeout_in_ms is 24 hrs >>>> 4. Disabled autocompaction on GCP node as this was heavily utilising >>>> CPU resource >>>> >>>> FYI, GCP rebuild process starts with data streaming from 3 nodes, and >>>> all fails one by one after streaming for a few hours. >>>> Please help out how to correct this issue. >>>> Is there any other way to rebuild such big data. >>>> We have a few tables with 200 - 400GB of data and some smaller tables. >>>> Also, we have Mviews in our environment >>>> >>>> Regards, >>>> Ashish Gupta >>>> >>>> >>>
Re: Streaming failure Issue
Logged "nodetool failuredetector" every 5sec. Doesn't seems to be an issue for phi_convict_threshold value On Tue, Oct 5, 2021 at 4:35 PM Surbhi Gupta wrote: > Hi , > > Try to adjust phi_convict_threshold and see if that helps. > When we did migration from on prim to AWS, this was one of the factor to > consider. > > Thanks > > > On Tue, Oct 5, 2021 at 4:00 AM MyWorld wrote: > >> Hi all, >> >> Need urgent help. >> We have one Physical Data Center of 5 nodes with 1 TB data on each >> (Location: Dallas). Currently we are using Cassandra ver 3.0.9. Now we are >> Adding one more Data Center of 5 nodes(Location GCP-US) and have joined it >> to the existing one. >> >> While running nodetool rebuild command, we are getting following error : >> On GCP node (where we ran rebuild command) : >> >>> ERROR [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,246 >>> StreamSession.java:639 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] >>> Remote peer 192.x.x.x failed stream session. >>> INFO [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,266 >>> StreamResultFuture.java:183 - [Stream >>> #66646d30-25a2-11ec-903b-774f88efe725] Session with /192.x.x.x is complete >> >> >> On DL source node : >> >>> INFO [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,785 >>> StreamResultFuture.java:183 - [Stream >>> #66646d30-25a2-11ec-903b-774f88efe725] Session with /34.x.x.x is complete >>> ERROR [STREAM-OUT-/34.x.x.x] 2021-10-05 15:55:53,785 >>> StreamSession.java:534 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] >>> Streaming error occurred >>> java.lang.RuntimeException: Transfer of file >>> /var/lib/cassandra/data/clickstream/glusr_usr_paid_url_mv-3c49c392b35511e9bd0a8f42dfb09617/mc-45676-big-Data.db >>> already completed or aborted (perhaps session failed?). >>> at >>> org.apache.cassandra.streaming.messages.OutgoingFileMessage.startTransfer(OutgoingFileMessage.java:120) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at >>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at >>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at >>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:48) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at >>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:387) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at >>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:367) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192] >>> WARN [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,786 >>> StreamResultFuture.java:210 - [Stream >>> #66646d30-25a2-11ec-903b-774f88efe725] Stream failed >> >> >> Before starting this rebuild, we have made the following changes: >> 1. Set setstreamthroughput to 600 Mb/sec >> 2. Set setinterdcstreamthroughput to 600 Mb/sec >> 3. streaming_socket_timeout_in_ms is 24 hrs >> 4. Disabled autocompaction on GCP node as this was heavily utilising CPU >> resource >> >> FYI, GCP rebuild process starts with data streaming from 3 nodes, and all >> fails one by one after streaming for a few hours. >> Please help out how to correct this issue. >> Is there any other way to rebuild such big data. >> We have a few tables with 200 - 400GB of data and some smaller tables. >> Also, we have Mviews in our environment >> >> Regards, >> Ashish Gupta >> >> >
Streaming failure Issue
Hi all, Need urgent help. We have one Physical Data Center of 5 nodes with 1 TB data on each (Location: Dallas). Currently we are using Cassandra ver 3.0.9. Now we are Adding one more Data Center of 5 nodes(Location GCP-US) and have joined it to the existing one. While running nodetool rebuild command, we are getting following error : On GCP node (where we ran rebuild command) : > ERROR [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,246 > StreamSession.java:639 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] > Remote peer 192.x.x.x failed stream session. > INFO [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,266 > StreamResultFuture.java:183 - [Stream > #66646d30-25a2-11ec-903b-774f88efe725] Session with /192.x.x.x is complete On DL source node : > INFO [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,785 > StreamResultFuture.java:183 - [Stream > #66646d30-25a2-11ec-903b-774f88efe725] Session with /34.x.x.x is complete > ERROR [STREAM-OUT-/34.x.x.x] 2021-10-05 15:55:53,785 > StreamSession.java:534 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] > Streaming error occurred > java.lang.RuntimeException: Transfer of file > /var/lib/cassandra/data/clickstream/glusr_usr_paid_url_mv-3c49c392b35511e9bd0a8f42dfb09617/mc-45676-big-Data.db > already completed or aborted (perhaps session failed?). > at > org.apache.cassandra.streaming.messages.OutgoingFileMessage.startTransfer(OutgoingFileMessage.java:120) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:48) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:387) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:367) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192] > WARN [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,786 > StreamResultFuture.java:210 - [Stream > #66646d30-25a2-11ec-903b-774f88efe725] Stream failed Before starting this rebuild, we have made the following changes: 1. Set setstreamthroughput to 600 Mb/sec 2. Set setinterdcstreamthroughput to 600 Mb/sec 3. streaming_socket_timeout_in_ms is 24 hrs 4. Disabled autocompaction on GCP node as this was heavily utilising CPU resource FYI, GCP rebuild process starts with data streaming from 3 nodes, and all fails one by one after streaming for a few hours. Please help out how to correct this issue. Is there any other way to rebuild such big data. We have a few tables with 200 - 400GB of data and some smaller tables. Also, we have Mviews in our environment Regards, Ashish Gupta
Data read size from table
Hi all, We are using Prometheus + grafana for monitoring apache cassandra with scrape interval of 15s. We have a table 'x' with partition size varying from 2mb to 80mb. We know there are few big partition entries present in this table and my objective is to monitor when this big partition entry is read from Cassandra(as it can be a cause of large GC pause) Now in Prometheus how can I figure out the "size of total data read" from table 'x' in last 15s. What formula can be applied? Regards, Ashish
Re: [UPGRADATION] Apache Cassandra from version 3.0.9 to 4.0.0
Thanks all for the help. On Mon, Sep 6, 2021 at 2:20 PM manish khandelwal < manishkhandelwa...@gmail.com> wrote: > Totally agree with Jeff and Bowen there. Don't try to achieve something > faster by cutting corners. Migration to GCP from physical DC should be done > on the same versions. > > On Mon, Sep 6, 2021 at 2:11 PM Bowen Song wrote: > >> Hello Ashish, >> >> >> I'm slightly worried about this: >> >> *Since I won't be needing physical DC anymore so instead of upgrading it >> I will simply discard that DC* >> >> This sounds like you are planning to add GCP 3.x to existing cluster, and >> upgrade GCP to 4.0, then decommission the existing DC without upgrading. If >> so, you need to think twice. Adding or removing nodes (or DCs) in a cluster >> with different versions is not recommended. I'd highly recommend you >> upgrade the existing DC before decommissioning it. Of course, you can skip >> the upgrade sstables on it which is often the most time consuming part. >> >> >> Cheers, >> >> Bowen >> >> On 06/09/2021 03:29, MyWorld wrote: >> >> Hi Jeff, >> >> *When you’re upgrading or rebuilding you want all copies on the same >> version with proper sstables . So either add GCP then upgrade to 4.0 or >> upgrade to 4.0 and then expand to GCP. Don’t do them at the same time.* >> >> I think I forgot to mention one thing that after completion of step 1 our >> GCP data center will be added with rebuild done on all nodes. So our >> complete cluster would be on 3.0.9 after step 1. Will change num_tokens >> from current 256 to 16 in GCP data center in this step only. >> >> DC1 - >> 5nodes (physical) - version 3.0.9 >> numtokens256 >> DC2 - >> 5nodes (GCP) - version 3.0.9 >> numtokens16 >> >> Rest all step from 2-5 are meant for upgradation in which I am planning >> to go DC wise upgradation and running upgradesstables on GCP first. >> >> DC1 - >> 5nodes (physical) - version 3.0.9 >> numtokens256 >> DC2 - >> 5nodes (GCP) - version 4.0.0 >> numtokens16 >> >> Since I won't be needing physical DC anymore so instead of upgrading it I >> will simply discard that DC >> >> Regards, >> Ashish >> >> On Mon, Sep 6, 2021, 7:31 AM Jeff Jirsa wrote: >> >>> In-line >>> >>> On Sep 3, 2021, at 11:12 AM, MyWorld wrote: >>> >>> >>> Hi Jeff, >>> Thanks for your response. >>> To answer your question, Yes, we have created dev environment by >>> restoring them from snapshot/CSV files. >>> >>> Just one follow up question, I have a 5-node single DC on production on >>> version 3.0.9on physical server. >>> We are planning to migrate to GCP along with upgradation using below >>> steps. >>> 1. Setup GCP data center with same version 3.0.9 and rebuild complete >>> data >>> 2. Now install and configure 4.0 version in new GCP data center on all 5 >>> nodes >>> 3. Stop version 3.0.9 and start 4.0 on all 5 nodes of GCP one by one >>> 4. Run upgradesstables one by one on all 5 nodes of GCP >>> 5.Later move read/write traffic to GCP and remove old datacenter which >>> is still on version 3.0.9 >>> >>> Please guide on few things: >>> 1. Is the above mention approach right? >>> >>> >>> When you’re upgrading or rebuilding you want all copies on the same >>> version with proper sstables . So either add GCP then upgrade to 4.0 or >>> upgrade to 4.0 and then expand to GCP. Don’t do them at the same time. >>> >>> >>> 2. OR should we update 4.0 on only one node on GCP at a time and run >>> upgrade sstables on just one node first >>> >>> >>> I usually do upgradesstables after all bounces are done >>> >>> The only exception is perhaps doing upgradesstables with exactly one >>> copy via backup/restore to make sure 4.0 works with your data files, which >>> it sounds like you’ve already done. >>> >>> 3. OR should we migrate to GCP first and then think of upgrade 4.0 later >>> 4. OR Is there any reason I should upgrade to 3.11.x first >>> >>> >>> Not 3.11 but maybe latest 3.0 instead >>> >>> >>> >>> Regards, >>> Ashish >>> >>> On Fri, Sep 3, 2021, 11:11 PM Jeff Jirsa wrote: >>> >>>> >>>> >>>> On Fri, Sep 3, 2021 at 10:33 AM MyWorld >>>> wrote: >
Re: [UPGRADATION] Apache Cassandra from version 3.0.9 to 4.0.0
Hi Jeff, *When you’re upgrading or rebuilding you want all copies on the same version with proper sstables . So either add GCP then upgrade to 4.0 or upgrade to 4.0 and then expand to GCP. Don’t do them at the same time.* I think I forgot to mention one thing that after completion of step 1 our GCP data center will be added with rebuild done on all nodes. So our complete cluster would be on 3.0.9 after step 1. Will change num_tokens from current 256 to 16 in GCP data center in this step only. DC1 - 5nodes (physical) - version 3.0.9 numtokens256 DC2 - 5nodes (GCP) - version 3.0.9 numtokens16 Rest all step from 2-5 are meant for upgradation in which I am planning to go DC wise upgradation and running upgradesstables on GCP first. DC1 - 5nodes (physical) - version 3.0.9 numtokens256 DC2 - 5nodes (GCP) - version 4.0.0 numtokens16 Since I won't be needing physical DC anymore so instead of upgrading it I will simply discard that DC Regards, Ashish On Mon, Sep 6, 2021, 7:31 AM Jeff Jirsa wrote: > In-line > > On Sep 3, 2021, at 11:12 AM, MyWorld wrote: > > > Hi Jeff, > Thanks for your response. > To answer your question, Yes, we have created dev environment by restoring > them from snapshot/CSV files. > > Just one follow up question, I have a 5-node single DC on production on > version 3.0.9on physical server. > We are planning to migrate to GCP along with upgradation using below steps. > 1. Setup GCP data center with same version 3.0.9 and rebuild complete data > 2. Now install and configure 4.0 version in new GCP data center on all 5 > nodes > 3. Stop version 3.0.9 and start 4.0 on all 5 nodes of GCP one by one > 4. Run upgradesstables one by one on all 5 nodes of GCP > 5.Later move read/write traffic to GCP and remove old datacenter which is > still on version 3.0.9 > > Please guide on few things: > 1. Is the above mention approach right? > > > When you’re upgrading or rebuilding you want all copies on the same > version with proper sstables . So either add GCP then upgrade to 4.0 or > upgrade to 4.0 and then expand to GCP. Don’t do them at the same time. > > > 2. OR should we update 4.0 on only one node on GCP at a time and run > upgrade sstables on just one node first > > > I usually do upgradesstables after all bounces are done > > The only exception is perhaps doing upgradesstables with exactly one copy > via backup/restore to make sure 4.0 works with your data files, which it > sounds like you’ve already done. > > 3. OR should we migrate to GCP first and then think of upgrade 4.0 later > 4. OR Is there any reason I should upgrade to 3.11.x first > > > Not 3.11 but maybe latest 3.0 instead > > > > Regards, > Ashish > > On Fri, Sep 3, 2021, 11:11 PM Jeff Jirsa wrote: > >> >> >> On Fri, Sep 3, 2021 at 10:33 AM MyWorld wrote: >> >>> Hi all, >>> We are doing a POC on dev environment to upgrade apache cassandra 3.0.9 >>> to 4.0.0. We have the below setup currently on cassandra 3.0.9 >>> DC1 - GCP(india) - 1 node >>> DC2 - GCP(US) - 1 node >>> >> >> 3.0.9 is very old. It's got older version of data files and some known >> correctness bugs. >> >> >>> >>> For upgradation, we carried out below steps on DC2 - GCP(US) node: >>> Step1. Install apache cassandra 4.0.0 >>> Step2. Did all Configuration settings >>> Step3. Stop apache cassandra 3.0.9 >>> Step4. Start apache cassandra 4.0.0 and monitor logs >>> Step5. Run nodetool upgradesstables and monitor logs >>> >>> After monitoring logs, I had below observations: >>> *1. Initially during bootstrap at Step4, received below exceptions:* >>> a) Exception (java.lang.IllegalArgumentException) encountered during >>> startup: Invalid sstable file manifest.json: the name doesn't look like a >>> supported sstable file name >>> java.lang.IllegalArgumentException: Invalid sstable file manifest.json: >>> the name doesn't look like a supported sstable file name >>> b) ERROR [main] 2021-08-29 06:25:52,120 CassandraDaemon.java:909 - >>> Exception encountered during startup >>> java.lang.IllegalArgumentException: Invalid sstable file schema.cql: the >>> name doesn't look like a supported sstable file name >>> >>> >> *In order to resolve, we removed manifest.json and schema.cql files from >>> each table directory and the issue was resolved. * >>> >> >> Did you restore these from backup/snapshot? >> >> >>> >>> *2. After resolving the above issue, we received below WARN messages >>> during bootstrap(step 4).* >>> *WARN * [main] 2021-08-29 0
Re: [UPGRADATION] Apache Cassandra from version 3.0.9 to 4.0.0
Hi Jeff, Thanks for your response. To answer your question, Yes, we have created dev environment by restoring them from snapshot/CSV files. Just one follow up question, I have a 5-node single DC on production on version 3.0.9on physical server. We are planning to migrate to GCP along with upgradation using below steps. 1. Setup GCP data center with same version 3.0.9 and rebuild complete data 2. Now install and configure 4.0 version in new GCP data center on all 5 nodes 3. Stop version 3.0.9 and start 4.0 on all 5 nodes of GCP one by one 4. Run upgradesstables one by one on all 5 nodes of GCP 5.Later move read/write traffic to GCP and remove old datacenter which is still on version 3.0.9 Please guide on few things: 1. Is the above mention approach right? 2. OR should we update 4.0 on only one node on GCP at a time and run upgrade sstables on just one node first 3. OR should we migrate to GCP first and then think of upgrade 4.0 later 4. OR Is there any reason I should upgrade to 3.11.x first Regards, Ashish On Fri, Sep 3, 2021, 11:11 PM Jeff Jirsa wrote: > > > On Fri, Sep 3, 2021 at 10:33 AM MyWorld wrote: > >> Hi all, >> We are doing a POC on dev environment to upgrade apache cassandra 3.0.9 >> to 4.0.0. We have the below setup currently on cassandra 3.0.9 >> DC1 - GCP(india) - 1 node >> DC2 - GCP(US) - 1 node >> > > 3.0.9 is very old. It's got older version of data files and some known > correctness bugs. > > >> >> For upgradation, we carried out below steps on DC2 - GCP(US) node: >> Step1. Install apache cassandra 4.0.0 >> Step2. Did all Configuration settings >> Step3. Stop apache cassandra 3.0.9 >> Step4. Start apache cassandra 4.0.0 and monitor logs >> Step5. Run nodetool upgradesstables and monitor logs >> >> After monitoring logs, I had below observations: >> *1. Initially during bootstrap at Step4, received below exceptions:* >> a) Exception (java.lang.IllegalArgumentException) encountered during >> startup: Invalid sstable file manifest.json: the name doesn't look like a >> supported sstable file name >> java.lang.IllegalArgumentException: Invalid sstable file manifest.json: >> the name doesn't look like a supported sstable file name >> b) ERROR [main] 2021-08-29 06:25:52,120 CassandraDaemon.java:909 - >> Exception encountered during startup >> java.lang.IllegalArgumentException: Invalid sstable file schema.cql: the >> name doesn't look like a supported sstable file name >> >> > *In order to resolve, we removed manifest.json and schema.cql files from >> each table directory and the issue was resolved. * >> > > Did you restore these from backup/snapshot? > > >> >> *2. After resolving the above issue, we received below WARN messages >> during bootstrap(step 4).* >> *WARN * [main] 2021-08-29 06:33:25,737 CommitLogReplayer.java:305 - >> Origin of 1 sstables is unknown or doesn't match the local node; >> commitLogIntervals for them were ignored >> *DEBUG *[main] 2021-08-29 06:33:25,737 CommitLogReplayer.java:306 - >> Ignored commitLogIntervals from the following sstables: >> [/opt1/cassandra_poc/data/clickstream/glcat_mcat_by_flname-af4e3ac0ace511ebaf9ec13e37d013c2/mc-1-big-Data.db] >> *WARN *[main] 2021-08-29 06:33:25,737 CommitLogReplayer.java:305 - >> Origin of 2 sstables is unknown or doesn't match the local node; >> commitLogIntervals for them were ignored >> *DEBUG *[main] 2021-08-29 06:33:25,738 CommitLogReplayer.java:306 - >> Ignored commitLogIntervals from the following sstables: >> [/opt1/cassandra_poc/data/clickstream/gl_city_map >> >> > Your data files dont match the commitlog files it expects to see. Either > you restored these from backup, or it's because 3.0.9 is much older than > 3.0.x that is more commonly used. > > >> *3. While upgrading sstables (step 5), we received below messages:* >> *WARN* [CompactionExecutor:3] 2021-08-29 07:47:32,828 >> DuplicateRowChecker.java:96 - Detected 2 duplicate rows for 29621439 during >> Upgrade sstables. >> *WARN* [CompactionExecutor:3] 2021-08-29 07:47:32,831 >> DuplicateRowChecker.java:96 - Detected 4 duplicate rows for 45016570 during >> Upgrade sstables. >> *WARN* [CompactionExecutor:3] 2021-08-29 07:47:32,833 >> DuplicateRowChecker.java:96 - Detected 3 duplicate rows for 61260692 during >> Upgrade sstables. >> >> > This says you have corrupt data from an old bug. Probably related to 2.1 > -> 3.0 upgrades, if this was originally on 2.1. If you read those keys, you > would find that the data returns 2-4 rows where it should be exactly 1. > > >> 4.* Also, received below messages during upgrade* >> *D
[UPGRADATION] Apache Cassandra from version 3.0.9 to 4.0.0
Hi all,
We are doing a POC on dev environment to upgrade apache cassandra 3.0.9 to
4.0.0. We have the below setup currently on cassandra 3.0.9
DC1 - GCP(india) - 1 node
DC2 - GCP(US) - 1 node
For upgradation, we carried out below steps on DC2 - GCP(US) node:
Step1. Install apache cassandra 4.0.0
Step2. Did all Configuration settings
Step3. Stop apache cassandra 3.0.9
Step4. Start apache cassandra 4.0.0 and monitor logs
Step5. Run nodetool upgradesstables and monitor logs
After monitoring logs, I had below observations:
*1. Initially during bootstrap at Step4, received below exceptions:*
a) Exception (java.lang.IllegalArgumentException) encountered during
startup: Invalid sstable file manifest.json: the name doesn't look like a
supported sstable file name
java.lang.IllegalArgumentException: Invalid sstable file manifest.json: the
name doesn't look like a supported sstable file name
b) ERROR [main] 2021-08-29 06:25:52,120 CassandraDaemon.java:909 -
Exception encountered during startup
java.lang.IllegalArgumentException: Invalid sstable file schema.cql: the
name doesn't look like a supported sstable file name
*In order to resolve, we removed manifest.json and schema.cql files from
each table directory and the issue was resolved. *
*2. After resolving the above issue, we received below WARN messages during
bootstrap(step 4).*
*WARN * [main] 2021-08-29 06:33:25,737 CommitLogReplayer.java:305 - Origin
of 1 sstables is unknown or doesn't match the local node;
commitLogIntervals for them were ignored
*DEBUG *[main] 2021-08-29 06:33:25,737 CommitLogReplayer.java:306 - Ignored
commitLogIntervals from the following sstables:
[/opt1/cassandra_poc/data/clickstream/glcat_mcat_by_flname-af4e3ac0ace511ebaf9ec13e37d013c2/mc-1-big-Data.db]
*WARN *[main] 2021-08-29 06:33:25,737 CommitLogReplayer.java:305 - Origin
of 2 sstables is unknown or doesn't match the local node;
commitLogIntervals for them were ignored
*DEBUG *[main] 2021-08-29 06:33:25,738 CommitLogReplayer.java:306 - Ignored
commitLogIntervals from the following sstables:
[/opt1/cassandra_poc/data/clickstream/gl_city_map
*3. While upgrading sstables (step 5), we received below messages:*
*WARN* [CompactionExecutor:3] 2021-08-29 07:47:32,828
DuplicateRowChecker.java:96 - Detected 2 duplicate rows for 29621439 during
Upgrade sstables.
*WARN* [CompactionExecutor:3] 2021-08-29 07:47:32,831
DuplicateRowChecker.java:96 - Detected 4 duplicate rows for 45016570 during
Upgrade sstables.
*WARN* [CompactionExecutor:3] 2021-08-29 07:47:32,833
DuplicateRowChecker.java:96 - Detected 3 duplicate rows for 61260692 during
Upgrade sstables.
4.* Also, received below messages during upgrade*
*DEBUG* [epollEventLoopGroup-5-8] 2021-09-03 12:27:31,347
InitialConnectionHandler.java:77 - OPTIONS received 5/v5
*DEBUG* [epollEventLoopGroup-5-8] 2021-09-03 12:27:31,349
InitialConnectionHandler.java:121 - Response to STARTUP sent, configuring
pipeline for 5/v5
*DEBUG* [epollEventLoopGroup-5-8] 2021-09-03 12:27:31,350
InitialConnectionHandler.java:153 - Configured pipeline:
DefaultChannelPipeline{(frameDecoder =
org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
*5. After upgrade, we are regularly getting below messages:*
*DEBUG* [ScheduledTasks:1] 2021-09-02 00:03:20,910 SSLFactory.java:354 -
Checking whether certificates have been updated []
*DEBUG* [ScheduledTasks:1] 2021-09-02 00:13:20,910 SSLFactory.java:354 -
Checking whether certificates have been updated []
*DEBUG* [ScheduledTasks:1] 2021-09-02 00:23:20,911 SSLFactory.java:354 -
Checking whether certificates have been updated []
*Can someone please explain what these above ERROR / WARN / DEBUG messages
refer to? Is there anything to be concerned about?*
*Also, received 2 READ_REQ dropped messages (may be due to nw latency) *
*INFO* [ScheduledTasks:1] 2021-09-03 11:40:10,009
MessagingMetrics.java:206 - READ_REQ messages were dropped in last 5000 ms:
0 internal and 1 cross node. Mean internal dropped latency: 0 ms and Mean
cross-node dropped latency: 12359 ms
*INFO* [ScheduledTasks:1] 2021-09-03 13:27:15,291
MessagingMetrics.java:206 - READ_REQ messages were dropped in last 5000 ms:
0 internal and 1 cross node. Mean internal dropped latency: 0 ms and Mean
cross-node dropped latency: 5960 ms
Rest of the stats are pretty much normal (tpstats, status, info,
tablestats, etc)
Regards,
Ashish
RE: RE: Re: Re: High mutation stage in multi dc deployment
Kindly help in this regard. What could be the possible reason for load and mutation spike in india data center On 2021/07/20 00:14:56 MyWorld wrote: > Hi Arvinder, > It's a separate cluster. Here max partition size is 32mb. > > On 2021/07/19 23:57:27 Arvinder Dhillon wrote: > > Is this the same cluster with 1G partition size? > > > > -Arvinder > > > > On Mon, Jul 19, 2021, 4:51 PM MyWorld wrote: > > > > > Hi daemeon, > > > We have already tuned the TCP settings to improve the bandwidth. Earlier > > > we had lot of hint and mutation msg drop which were gone after tuning > TCP. > > > Moreover we are writing with CL local quorum at US side, so ack is taken > > > from local DC. > > > I m still concern what could be reason of increase mutation count. > > > > > > On 2021/07/19 19:55:52 daemeon reiydelle wrote: > > > > You may want to think about the latency impacts of a cluster that has > one > > > > node "far away". This is such a basic design flaw that you need to do > > > some > > > > basic learning, and some basic understanding of networking and > latency. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jul 19, 2021 at 10:38 AM MyWorld wrote: > > > > > > > > > Hi all, > > > > > > > > > > Currently we have a cluster with 2 DC of 3 nodes each. One DC is in > > > GCP-US > > > > > while other is in GCP-India. Just to add here, configuration of > every > > > node > > > > > accross both DC is same. Cpu-6, Ram-32gb, Heap-8gb > > > > > > > > > > We do all our write on US data center. While performing a bulk > write on > > > > > GCP US, we observe normal load of 1 on US while this load at GCP > India > > > > > spikes to 10. > > > > > > > > > > On observing tpstats further in grafana we found mutation stage at > GCP > > > > > India is going to 1million intermittently though our overall write > is > > > > > nearly 300 per sec per node. Don't know the reason but whenever we > have > > > > > this spike, we are having load issue. > > > > > Please help what could be the possible reason for this? > > > > > > > > > > Regards, > > > > > Ashish > > > > > > > > > > > > > > >
RE: RE: Storing user activity logs
Thanks Elliott, Sean, Manish for the quick response. On 2021/07/20 13:31:17 "Durity, Sean R" wrote: > Yes, use the time-bucketing approach and choose a bucket-size (included in the partition key) that is granular enough to keep partitions to about 100 MB in size. (Unbounded partitions WILL destroy your cluster.) If your queries *need* to retrieve all user activity over a certain period, then, yes, multiple queries may be required. Partition key queries (of small partitions) are very fast and can be done asynchronously. That is the right way to use Cassandra for a time series of data. > > Sean Durity – Staff Systems Engineer, Cassandra > > From: manish khandelwal > Sent: Monday, July 19, 2021 11:58 PM > To: user@cassandra.apache.org > Subject: [EXTERNAL] Re: Storing user activity logs > > I concur with Eliot view. Only way you can reduce partition size is by tweaking your partition key. Here with user_id as partition key, partition size depends on the activity of the user. For a superactive user it can become large in no time. After changing the key migration of old data to the new table will also be required, please keep that also in mind. > > Regards > Manish > > On Tue, Jul 20, 2021 at 2:54 AM Elliott Sims > wrote: > Your partition key determines your partition size. Reducing retention sounds like it would help some in your case, but really you'd have to split it up somehow. If it fits your query pattern, you could potentially have a compound key of userid+datetime, or some other time-based split. You could also just split each user's rows into subsets with some sort of indirect mapping, though that can get messy pretty fast. > > On Mon, Jul 19, 2021 at 9:01 AM MyWorld > wrote: > Hi all, > > We are currently storing our user activity log in Cassandra with below architecture. > > Create table user_act_log( > Userid bigint, > Datetime bigint, > Sno UUID, > some more columns) > With partition key - userid > Clustering key - datetime, sno > And TTL of 6 months > > With time our table data have grown to around 500gb and we notice from table histogram our max partition size have also grown to tremendous size (nearly 1gb) > > So, please help me out what should be the right architecture for this use case? > > I am currently thinking of changing the compaction strategy to time window from size tier with 30 day window. But will this improve the partion size? > > Should we use any other db for such use case? > > > > > > > The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment. >
RE: Re: Re: High mutation stage in multi dc deployment
Hi Arvinder, It's a separate cluster. Here max partition size is 32mb. On 2021/07/19 23:57:27 Arvinder Dhillon wrote: > Is this the same cluster with 1G partition size? > > -Arvinder > > On Mon, Jul 19, 2021, 4:51 PM MyWorld wrote: > > > Hi daemeon, > > We have already tuned the TCP settings to improve the bandwidth. Earlier > > we had lot of hint and mutation msg drop which were gone after tuning TCP. > > Moreover we are writing with CL local quorum at US side, so ack is taken > > from local DC. > > I m still concern what could be reason of increase mutation count. > > > > On 2021/07/19 19:55:52 daemeon reiydelle wrote: > > > You may want to think about the latency impacts of a cluster that has one > > > node "far away". This is such a basic design flaw that you need to do > > some > > > basic learning, and some basic understanding of networking and latency. > > > > > > > > > > > > > > > > > > On Mon, Jul 19, 2021 at 10:38 AM MyWorld wrote: > > > > > > > Hi all, > > > > > > > > Currently we have a cluster with 2 DC of 3 nodes each. One DC is in > > GCP-US > > > > while other is in GCP-India. Just to add here, configuration of every > > node > > > > accross both DC is same. Cpu-6, Ram-32gb, Heap-8gb > > > > > > > > We do all our write on US data center. While performing a bulk write on > > > > GCP US, we observe normal load of 1 on US while this load at GCP India > > > > spikes to 10. > > > > > > > > On observing tpstats further in grafana we found mutation stage at GCP > > > > India is going to 1million intermittently though our overall write is > > > > nearly 300 per sec per node. Don't know the reason but whenever we have > > > > this spike, we are having load issue. > > > > Please help what could be the possible reason for this? > > > > > > > > Regards, > > > > Ashish > > > > > > > > > >
RE: Re: Storing user activity logs
Hi Elliott, We thought of adding month and mapping key to partition key to make it bimonthly. So our new partition key would be userid + date (mm) + mapping key (01 for date 01-15 and 02 for date 16-30). However, there could be a user who has done only 10 activities in past 6 months. So I need to do 12 reads to find his all activity logs and there could be another user who has done enormous activities within a month. I don't think doing further granular partitioning would help as read queries will increase. And what if the user activity log for some users after bimonthly partitioning increased to 500mb . On 2021/07/19 21:23:24 Elliott Sims wrote: > Your partition key determines your partition size. Reducing retention > sounds like it would help some in your case, but really you'd have to split > it up somehow. If it fits your query pattern, you could potentially have a > compound key of userid+datetime, or some other time-based split. You could > also just split each user's rows into subsets with some sort of indirect > mapping, though that can get messy pretty fast. > > On Mon, Jul 19, 2021 at 9:01 AM MyWorld wrote: > > > Hi all, > > > > We are currently storing our user activity log in Cassandra with below > > architecture. > > > > Create table user_act_log( > > Userid bigint, > > Datetime bigint, > > Sno UUID, > > some more columns) > > With partition key - userid > > Clustering key - datetime, sno > > And TTL of 6 months > > > > With time our table data have grown to around 500gb and we notice from > > table histogram our max partition size have also grown to tremendous size > > (nearly 1gb) > > > > So, please help me out what should be the right architecture for this use > > case? > > > > I am currently thinking of changing the compaction strategy to time window > > from size tier with 30 day window. But will this improve the partion size? > > > > Should we use any other db for such use case? > > > > > > > > >
RE: Re: High mutation stage in multi dc deployment
Hi daemeon, We have already tuned the TCP settings to improve the bandwidth. Earlier we had lot of hint and mutation msg drop which were gone after tuning TCP. Moreover we are writing with CL local quorum at US side, so ack is taken from local DC. I m still concern what could be reason of increase mutation count. On 2021/07/19 19:55:52 daemeon reiydelle wrote: > You may want to think about the latency impacts of a cluster that has one > node "far away". This is such a basic design flaw that you need to do some > basic learning, and some basic understanding of networking and latency. > > > > > > On Mon, Jul 19, 2021 at 10:38 AM MyWorld wrote: > > > Hi all, > > > > Currently we have a cluster with 2 DC of 3 nodes each. One DC is in GCP-US > > while other is in GCP-India. Just to add here, configuration of every node > > accross both DC is same. Cpu-6, Ram-32gb, Heap-8gb > > > > We do all our write on US data center. While performing a bulk write on > > GCP US, we observe normal load of 1 on US while this load at GCP India > > spikes to 10. > > > > On observing tpstats further in grafana we found mutation stage at GCP > > India is going to 1million intermittently though our overall write is > > nearly 300 per sec per node. Don't know the reason but whenever we have > > this spike, we are having load issue. > > Please help what could be the possible reason for this? > > > > Regards, > > Ashish > > >
RE: Re: High mutation stage in multi dc deployment
Hi Patrick, Currently we are using 3.11.6 apache cassandra version. We are performing write with CL local quorum in US side DC. We have 4-5 tables with supplier_details, supplier_prod_details, supplier_rating. We also have an mview attached with rating table. For batching part, I need to check with ops team. However they are re-syncing data supplier wise in these tables. On 2021/07/19 20:56:49 Patrick McFadin wrote: > Hi Ashish, > > Can you give us some information about some of the details? Specifically > some indication on the version of Cassandra, data model, consistency levels > used, and how you are bulk loading. Is this a batch by any chance? > > Patrick > > On Mon, Jul 19, 2021 at 10:38 AM MyWorld wrote: > > > Hi all, > > > > Currently we have a cluster with 2 DC of 3 nodes each. One DC is in GCP-US > > while other is in GCP-India. Just to add here, configuration of every node > > accross both DC is same. Cpu-6, Ram-32gb, Heap-8gb > > > > We do all our write on US data center. While performing a bulk write on > > GCP US, we observe normal load of 1 on US while this load at GCP India > > spikes to 10. > > > > On observing tpstats further in grafana we found mutation stage at GCP > > India is going to 1million intermittently though our overall write is > > nearly 300 per sec per node. Don't know the reason but whenever we have > > this spike, we are having load issue. > > Please help what could be the possible reason for this? > > > > Regards, > > Ashish > > >
RE: Re: R/W timeouts VS number of tables in keyspace
Yes it seems like a GC issue. And since the default timeout for write/read is 2000ms and 5000ms, you might be experiencing timeout issues. But you need to check reason behind this GC pause. Heavy partition or high tombstones could be one reason. Check your table stats for the same. As suggested you can opt for g1gc On 2021/07/19 16:21:02 Luca Rondanini wrote: > Thanks Yakir, > > I can already experience slow repairs and startups but I'd like to > stabilize the system before jumping into refactoring (columns are not a > problem, max 10/cols per table). Do you believe it's a GC problem to cause > the timeouts and crashes? I'll give it a try and update this post. > > Thanks, > Luca > > > > On Mon, Jul 19, 2021 at 9:14 AM Yakir Gibraltar wrote: > > > I recommend rethinking about this design, hard to maintain, slow startup > > and repair . > > About GC, try to replace CMS with G1 , see doc : > > https://docs.datastax.com/en/dse/6.0/dse-admin/datastax_enterprise/operations/opsTuningGcAbout.html > > BTW, also many columns may affect performance, see doc: > > https://thelastpickle.com/blog/2020/12/17/impacts-of-many-columns-in-cassandra-table.html > > > > Cheers, Yakir Gibraltar > > > > >
High mutation stage in multi dc deployment
Hi all, Currently we have a cluster with 2 DC of 3 nodes each. One DC is in GCP-US while other is in GCP-India. Just to add here, configuration of every node accross both DC is same. Cpu-6, Ram-32gb, Heap-8gb We do all our write on US data center. While performing a bulk write on GCP US, we observe normal load of 1 on US while this load at GCP India spikes to 10. On observing tpstats further in grafana we found mutation stage at GCP India is going to 1million intermittently though our overall write is nearly 300 per sec per node. Don't know the reason but whenever we have this spike, we are having load issue. Please help what could be the possible reason for this? Regards, Ashish
Storing user activity logs
Hi all, We are currently storing our user activity log in Cassandra with below architecture. Create table user_act_log( Userid bigint, Datetime bigint, Sno UUID, some more columns) With partition key - userid Clustering key - datetime, sno And TTL of 6 months With time our table data have grown to around 500gb and we notice from table histogram our max partition size have also grown to tremendous size (nearly 1gb) So, please help me out what should be the right architecture for this use case? I am currently thinking of changing the compaction strategy to time window from size tier with 30 day window. But will this improve the partion size? Should we use any other db for such use case?
Re: tablehistogram shows high sstables
High latency at table level could be a result of no of factors. High tombstones GC Bloom filter fp I/O bottleneck Insufficient cache or threads Since your partion size is not a problem here, try running major compaction on this table. Also you can turn on caching for this table. As mentioned above check other factors and eliminate one by one. Regards, Ashish On Sat 1 May, 2021, 06:41 vytenis silgalis, wrote: > 17ms read latency for the 50th percentile is actually a pretty high > latency in my experience, I prefer to see the 75th percentile read latency > to be around 1-2ms. Of course it depends on your use case and what your > performance objectives are. > > On Thu, Apr 29, 2021 at 7:05 AM Kane Wilson wrote: > >> It does imply the SSTables are being read - how big is your data size and >> how much memory on the nodes? It's certainly possible to get low latencies >> despite many SSTables, but I'd expect small read sizes paired with a lot of >> memory. >> >> >> raft.so - Cassandra consulting, support, managed services >> >> On Thu., 29 Apr. 2021, 08:44 Ayub M, wrote: >> >>> The table has 24 sstables with size tiered compaction, when I run >>> nodetool tablehistograms I see 99% percentile of the queries are showing up >>> 24 as the number of sstables. But the read latency is very low, my >>> understanding from the tableshistograms's sstable column is - it shows how >>> many sstables were read to complete the query. If so reading 24 sstables >>> should take sometime, at least maybe couple of seconds. Am I missing >>> something here? Does checking against index/bloom filters count towards >>> sstable counter as well? >>> >>> Percentile SSTables Write Latency Read LatencyPartition Size >>> Cell Count >>> (micros) (micros) (bytes) >>> 50%24.00 17.08 17436.92 310 >>>6 >>> 75%24.00 24.60 20924.30 446 >>>6 >>> 95%24.00 42.51 62479.63 770 >>> 10 >>> 98%24.00 51.01 74975.55 1597 >>> 17 >>> 99%24.00 61.21 74975.55 3311 >>> 24 >>> Min18.00 2.30 4866.3287 >>>0 >>> Max24.00943.13 89970.66545791 >>>17084 >>> >>>
Re: Getting error: "no connections were made when creating the session"
Thanks Kane. But go client connection settings contain only the public address of a single live node. Other nodes were already removed from the connection string. Moreover, we get the server reboot in order to clear any cache. Still the error was the same : "no connections were made when creating the session". I completely agree with your point : "Neither of those changes should have had any effect on client connections, as they only relate to internode communication." But somehow only those settings change corrected the issue. Still have no clue howcome? On Fri, May 7, 2021 at 4:45 AM Kane Wilson wrote: > Neither of those changes should have had any effect on client connections, > as they only relate to internode communication. Your go client should have > specified the public ip addresses as per the broadcast RPC address, and > would need to be specifying the live node as the connection point to work. > I suspect you encountered a situation where the addresses specified in your > go client were the decommissioned nodes and maybe it didn't attempt all > nodes or the live node was initially missing from that list? > > > raft.so - Cassandra consulting, support, and managed services > > > On Thu, May 6, 2021 at 11:35 PM MyWorld wrote: > >> Hi All, >> >> We had a set up of 3 node single DC cluster on Google cloud with RF=3 for >> development purposes. Service Team hosted in Dallas servers was able to >> connect from php as well as golang code properly. Initially we have set up >> the following settings in cassandra.yaml: >> Listen address: >> Broadcast address: >> Seeds: >> Rpc address: 0.0.0.0 >> Broadcast rpc address: >> >> Due to cost factor, we removed 2 nodes from our DC and now we had a >> single node DC cluster with RF=1 (therefore no peers). >> >> After rebooting the servers(in Dallas) at service end, php code was able >> to connect properly. But from go lang service, we started getting the >> following error: "no connections were made when creating the session". We >> firstly set DisableInitialHostLookup to true,but that didn't resolve the >> issue. When we make below changes in cassandra.yaml ,this issue was >> resolved: >> Broadcast address: >> Seeds: >> >> Now, not able to understand: >> 1. Why was there no issue when we had 3 node DC set up with the broadcast >> address set to PrivateIP? >> 2. How does change in Broadcast address from private to public has an >> impact on client connection for single node DC since we already had rpc >> broadcast address set to public for remote clients. >> >> Please help me understand this. >> >> Regards, >> Ashish Gupta >> >
Getting error: "no connections were made when creating the session"
Hi All, We had a set up of 3 node single DC cluster on Google cloud with RF=3 for development purposes. Service Team hosted in Dallas servers was able to connect from php as well as golang code properly. Initially we have set up the following settings in cassandra.yaml: Listen address: Broadcast address: Seeds: Rpc address: 0.0.0.0 Broadcast rpc address: Due to cost factor, we removed 2 nodes from our DC and now we had a single node DC cluster with RF=1 (therefore no peers). After rebooting the servers(in Dallas) at service end, php code was able to connect properly. But from go lang service, we started getting the following error: "no connections were made when creating the session". We firstly set DisableInitialHostLookup to true,but that didn't resolve the issue. When we make below changes in cassandra.yaml ,this issue was resolved: Broadcast address: Seeds: Now, not able to understand: 1. Why was there no issue when we had 3 node DC set up with the broadcast address set to PrivateIP? 2. How does change in Broadcast address from private to public has an impact on client connection for single node DC since we already had rpc broadcast address set to public for remote clients. Please help me understand this. Regards, Ashish Gupta
Setting DC in different geographical location
Hi, We have a cluster with one Data Center of 3 nodes in GCP-US(RF=3).Current apache cassandra version 3.11.6. We are planning to add one new Data Center of 3 nodes in GCP-India. At peak hours, files generation in commit logs at GCP-US side on one node is around 1 GB per minute (i.e 17+ mbps). Currently the file transfer speed from GCP US to India is 9 mbps. So, with this speed, is it possible in cassandra to perform asynchronous write in new DC(India)? Also, is there any compression technique which cassandra applies while transferring data across DC? *My assumption *: All 3 coordinator nodes in US will be responsible for transfering 1/3rd data to new DC. So, at peak time only 1GB/3 is what each node has to sync. Please let me know is my assumption right? If yes, what will happen if data generated in commit log per node increase to 3 GB per minute tomorrow. Regards, Ashish
Re: Cassandra timeouts 3.11.6
Thanks Sean, will definitely check this point. Also, query tracing is allowed from version 4.0. Is there any way to trace queries in my current version 3.11.6 On Tue 26 Jan, 2021, 20:01 Durity, Sean R (US), wrote: > I would be looking at the queries in the application to see if there are > any cross-partition queries (ALLOW FILTERING or IN clauses across > partitions). This looks like queries that work fine with small scale, but > are hitting timeouts when the data size has increased. > > Also see if anyone has ad-hoc access to the cluster and is running some of > those same kind of bad queries. > > > > Sean Durity > > > > *From:* MyWorld > *Sent:* Tuesday, January 26, 2021 8:50 AM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] Cassandra timeouts 3.11.6 > > > > Hi, > > We have a cluster of 4 nodes all in one DC (apache cass version : 3.11.6). > > Things were working fine till last month when all of a sudden we started > facing Operation time outs at client end intermittently. > > > > We have prometheus+grafana configured for monitoring. > > On checking we found the following points: > > 1. Read/Write latency at coordinator level increase at the same time on > one or multiple nodes. > > 2. jvm_threads_current increases at the same time on one or multiple > nodes. > > 3. Cassandra hints storage increases at the same time on one or multiple > nodes. > > 4. Increase in Client Requests + Connection timeout with dropped messages > at times. > > 5. Increase in connected native clients count on all nodes > > > > Things already checked : > > 1. No change in read/write requests. > > 2. No major change in table level read/write latency. > > 3. No spike on DB load or CPU utilization. > > 4. Memory usage is also normal. > > 5. GC Pauses are also normal. > > 6. No packet loss between nodes at network level > > > > On checking detailed logs, we found majorly two types of messages during > timeouts. > > 1. HintsDispatcher : Dispatching hints from one node to other > > 2. READ messages were dropped in last 5000 ms: 0 internal and 1 cross > node. Mean internal dropped latency: 0 ms and Mean cross-node dropped > latency: 14831 ms > > 3. StatusLogger messages. > > > > Please suggest possible reasons for the same and action items. > > > > Regards, > > Ashish > > -- > > The information in this Internet Email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this Email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. When addressed > to our clients any opinions or advice contained in this Email are subject > to the terms and conditions expressed in any applicable governing The Home > Depot terms of business or client engagement letter. The Home Depot > disclaims all responsibility and liability for the accuracy and content of > this attachment and for any damages or losses arising from any > inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other > items of a destructive nature, which may be contained in this attachment > and shall not be liable for direct, indirect, consequential or special > damages in connection with this e-mail message or its attachment. >
Cassandra timeouts 3.11.6
Hi, We have a cluster of 4 nodes all in one DC (apache cass version : 3.11.6). Things were working fine till last month when all of a sudden we started facing Operation time outs at client end intermittently. We have prometheus+grafana configured for monitoring. On checking we found the following points: 1. Read/Write latency at coordinator level increase at the same time on one or multiple nodes. 2. jvm_threads_current increases at the same time on one or multiple nodes. 3. Cassandra hints storage increases at the same time on one or multiple nodes. 4. Increase in Client Requests + Connection timeout with dropped messages at times. 5. Increase in connected native clients count on all nodes Things already checked : 1. No change in read/write requests. 2. No major change in table level read/write latency. 3. No spike on DB load or CPU utilization. 4. Memory usage is also normal. 5. GC Pauses are also normal. 6. No packet loss between nodes at network level On checking detailed logs, we found majorly two types of messages during timeouts. 1. HintsDispatcher : Dispatching hints from one node to other 2. READ messages were dropped in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 14831 ms 3. StatusLogger messages. Please suggest possible reasons for the same and action items. Regards, Ashish
Node configuration and capacity
Hi, We are currently using apache cassandra 3.11.6 in our production environment with single DC of 4 nodes. 2 nodes have configuration : Ssd 24 cores, 64gb ram, 20gb heap size Other 2 nodes have: Ssd 32cores, 64gb ram, 20gb heap size I have several questions around this. 1. Does different configuration nodes(cores) in single dc have any impact ? 2. Can we have different heap size in single DC on different nodes? 3. Which is better : single partition disk or multiple partition disk? 4. Currently we have 200 writes and around 5000 reads per sec per node (In 4 node cluster). How to determine max node capacity? 5. We are getting read/write operation timeout intermittently. There is no GC issue. However we have observed 1% packet loss between nodes. Can this be the cause of timeout issue? 6. Currently we are getting 1100 established connections from client side. Shall we increase native_transport_max_threads to 1000+? Currently we have increased it from default 128 to 512 after finding pending NTR requests during timeout issue. 7. Have found below h/w production recommendation from dse site. How much this is helpful for apache cassandra ? net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.core.rmem_default=16777216 net.core.wmem_default=16777216 net.core.optmem_max=40960 net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216
Operation timeout error from Apache Cassandra 3.11.6
Hi, We are currently using apache cassandra 3.11.6. We have a cluster of 4 nodes all in single DC. All of a sudden we started experiencing high connection timeout on the cluster. This timeout spikes are coming intermittently. As checked from grafana, read/write time, jvm threads count, client counts got increased at the same time. But there isn't any major GC pause at this time. *On client side* : Using Local Quorum read/write and getting error Operation timed out - received only 1 responses *On checking debug.log:* DEBUG [Native-Transport-Requests-39] 2020-12-23 15:05:26,843 ReadCallback.java:132 - Timed out; received 2 of 3 responses (including data) DEBUG [Native-Transport-Requests-16] 2020-12-23 15:05:27,016 ReadCallback.java:132 - Timed out; received 2 of 3 responses (including data) DEBUG [Native-Transport-Requests-47] 2020-12-23 15:05:30,627 ReadCallback.java:132 - Timed out; received 0 of 1 responses DEBUG [Native-Transport-Requests-35] 2020-12-23 15:05:31,163 ReadCallback.java:132 - Timed out; received 2 of 3 responses (including data) DEBUG [Native-Transport-Requests-54] 2020-12-23 15:05:31,481 ReadCallback.java:132 - Timed out; received 2 of 3 responses (including data) DEBUG [Native-Transport-Requests-7] 2020-12-23 15:05:33,740 ReadCallback.java:132 - Timed out; received 2 of 3 responses (including data) DEBUG [Native-Transport-Requests-8] 2020-12-23 15:05:37,479 ReadCallback.java:132 - Timed out; received 2 of 3 responses (including data) INFO [ScheduledTasks:1] 2020-12-23 15:05:43,287 StatusLogger.java:47 - Pool NameActive Pending Completed Blocked All Time Blocked INFO [ScheduledTasks:1] 2020-12-23 15:05:43,295 StatusLogger.java:51 - MutationStage 0 0 19306477 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,295 StatusLogger.java:51 - ViewMutationStage 0 0 0 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,295 StatusLogger.java:51 - ReadStage 1 0 338493026 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,296 StatusLogger.java:51 - RequestResponseStage 0 0 181164440 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,296 StatusLogger.java:51 - ReadRepairStage 0 0 90518075 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,296 StatusLogger.java:51 - CounterMutationStage 0 0 0 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,297 StatusLogger.java:51 - MiscStage 0 0 0 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,297 StatusLogger.java:51 - CompactionExecutor0 0 390126 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,297 StatusLogger.java:51 - MemtableReclaimMemory 0 0335 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,297 StatusLogger.java:51 - PendingRangeCalculator0 0 6 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,298 StatusLogger.java:51 - GossipStage 0 0 479616 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,298 StatusLogger.java:51 - SecondaryIndexManagement 0 0 0 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,298 StatusLogger.java:51 - HintsDispatcher 0 0242 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,299 StatusLogger.java:51 - MigrationStage0 0 7 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,299 StatusLogger.java:51 - MemtablePostFlush 0 0367 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,299 StatusLogger.java:51 - PerDiskMemtableFlushWriter_0 0 0335 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,300 StatusLogger.java:51 - ValidationExecutor0 0 19 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,300 StatusLogger.java:51 - Sampler 0 0 0 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,300 StatusLogger.java:51 - MemtableFlushWriter 0 0335 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,300 StatusLogger.java:51 - InternalResponseStage 0 0 464309 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,301 StatusLogger.java:51 - AntiEntropyStage 0 0 70 0 0 INFO [ScheduledTasks:1] 2020-12-23 15:05:43,301
