Re: Maturity and Stability of Enabling CDC
Thanks Jeff! On Mon, Sep 18, 2017 at 9:31 AM, Jeff Jirsa <jji...@gmail.com> wrote: > Haven't tried out CDC, but the answer based on the design doc is yes - you > have to manually dedup CDC at the consumer level > > > > > -- > Jeff Jirsa > > > On Sep 17, 2017, at 6:21 PM, Michael Fong <mcfong.o...@gmail.com> wrote: > > Thanks for your reply. > > > If anyone has tried out this new feature, perhaps he/she could answer this > question: would multiple copies of CDC be sent to downstream (i.e., Kafka) > when all nodes have enabled cdc? > > Regards, > > On Mon, Sep 18, 2017 at 6:59 AM, kurt greaves <k...@instaclustr.com> > wrote: > >> I don't believe it's used by many, if any. it certainly hasn't had enough >> attention to determine it production ready, nor has it been out long enough >> for many people to be in a version where cdc is available. FWIW I've never >> even seen any inquiries about using it. >> >> On 17 Sep. 2017 13:18, "Michael Fong" <mcfong.o...@gmail.com> wrote: >> >> anyone? >> >> On Wed, Sep 13, 2017 at 5:10 PM, Michael Fong <mcfong.o...@gmail.com> >> wrote: >> >>> Hi, all, >>> >>> We've noticed there is a new feature for streaming changed data other >>> streaming service. Doc: http://cassandra.apache.o >>> rg/doc/latest/operating/cdc.html >>> >>> We are evaluating the stability (and maturity) of this feature, and >>> possibly integrate this with Kafka (associated w/ its connector). Has >>> anyone adopted this in production for real application? >>> >>> Regards, >>> >>> Michael >>> >> >> >> >
Re: Maturity and Stability of Enabling CDC
Thanks for your reply. If anyone has tried out this new feature, perhaps he/she could answer this question: would multiple copies of CDC be sent to downstream (i.e., Kafka) when all nodes have enabled cdc? Regards, On Mon, Sep 18, 2017 at 6:59 AM, kurt greaves <k...@instaclustr.com> wrote: > I don't believe it's used by many, if any. it certainly hasn't had enough > attention to determine it production ready, nor has it been out long enough > for many people to be in a version where cdc is available. FWIW I've never > even seen any inquiries about using it. > > On 17 Sep. 2017 13:18, "Michael Fong" <mcfong.o...@gmail.com> wrote: > > anyone? > > On Wed, Sep 13, 2017 at 5:10 PM, Michael Fong <mcfong.o...@gmail.com> > wrote: > >> Hi, all, >> >> We've noticed there is a new feature for streaming changed data other >> streaming service. Doc: http://cassandra.apache.org/doc/latest/operating/ >> cdc.html >> >> We are evaluating the stability (and maturity) of this feature, and >> possibly integrate this with Kafka (associated w/ its connector). Has >> anyone adopted this in production for real application? >> >> Regards, >> >> Michael >> > > >
Re: Maturity and Stability of Enabling CDC
anyone? On Wed, Sep 13, 2017 at 5:10 PM, Michael Fong <mcfong.o...@gmail.com> wrote: > Hi, all, > > We've noticed there is a new feature for streaming changed data other > streaming service. Doc: http://cassandra.apache.org/doc/latest/operating/ > cdc.html > > We are evaluating the stability (and maturity) of this feature, and > possibly integrate this with Kafka (associated w/ its connector). Has > anyone adopted this in production for real application? > > Regards, > > Michael >
Maturity and Stability of Enabling CDC
Hi, all, We've noticed there is a new feature for streaming changed data other streaming service. Doc: http://cassandra.apache.org/doc/latest/operating/cdc .html We are evaluating the stability (and maturity) of this feature, and possibly integrate this with Kafka (associated w/ its connector). Has anyone adopted this in production for real application? Regards, Michael
Is it possible to recover a deleted-in-future record?
Hi, all, We recently encountered an issue in production that some records were mysteriously deleted with a timestamp 100+ years from now. Everything is normal as of now, and how the deletion happened and accuracy of system timestamp at that moment are unknown. We were wondering if there is a general way to recover the mysteriously-deleted data when the timestamp meta is screwed up. Thanks in advanced, Regards, Michael Fong
Schema Disagreement vs Nodetool resetlocalschema
Hi, We have recently encountered several schema disagreement issue while upgrading Cassandra. In one of the cases, the 2-node cluster idled for over 30 minutes and their schema remain unsynced. Due to other logic flows, Cassandra cannot be restarted, and hence we need to come up an alternative on-the-fly. We are thinking to do a nodetool resetlocalschema to force the schema synchronization. How safe is this method? Do we need to disable thrift/gossip protocol before performing this function, and enable them back after resync completes? Thanks in advance! Sincerely, Michael Fong
Gossip Behavioral Difference between C* 2.0 and C* 2.1
Hi, We recently discovered that there are some differences in gossip behavior between C* 2.0 and C* 2.1. In some cases of network instability or a node reboot, we can observe some behavioral differences from Cassandra/system.log. 2.0.17 We can observe this log of similar pattern in log : DEBUG [RequestResponseStage:3] 2016-04-19 11:18:18,332 Gossiper.java (line 977) removing expire time for endpoint : /192.168.88.34 INFO [RequestResponseStage:3] 2016-04-19 11:18:18,333 Gossiper.java (line 978) InetAddress /192.168.88.34 is now UP DEBUG [RequestResponseStage:4] 2016-04-19 11:18:18,335 Gossiper.java (line 977) removing expire time for endpoint : /192.168.88.34 INFO [RequestResponseStage:4] 2016-04-19 11:18:18,335 Gossiper.java (line 978) InetAddress /192.168.88.34 is now UP DEBUG [RequestResponseStage:3] 2016-04-19 11:18:18,335 Gossiper.java (line 977) removing expire time for endpoint : /192.168.88.34 INFO [RequestResponseStage:3] 2016-04-19 11:18:18,335 Gossiper.java (line 978) InetAddress /192.168.88.34 is now UP It seems the longer for the node to regain connection (or reboot), the more accumulated gossip message, and the more gossip message will appear afterwards. However, in 2.1, we do not observe this kind of behavior any more. There seems to be some fundamental changes on gossip protocol. Did anyone also observe the similar pattern, or could kindly point out which changes (JIRA #) that made of this improvement? Thanks in advanced! Sincerely, Michael Fong
RE: Cassandra 2.0.x OOM during startsup - schema version inconsistency after reboot
Hi Alain, Thanks for your reply. We understood that there is a chance that this would be left unresolved, since we are really way behind the official Cassandra releases. Here is what have further found for the OOM issue, which seems to be related to # of gossip message accumulated on a live node waiting to connect to the rebooted node. Once that node is rebooted, all the gossip message floods into each other, triggers StorageService.onAlive() and schedules a schema pull on demand. In our case, schema version sometimes is different after reboot. When that happens, schema-exchange storm begins. Also, thanks for your tip on sharing the SOP on stopping an ode, here is what we have for our stop procedure: Disable thrift Disable Binary Wait 10s Disable gossip Drain Kill Any thought on this to be further improved? Thanks! Sincerely, Michael Fong From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Wednesday, May 11, 2016 10:01 PM To: user@cassandra.apache.org Cc: d...@cassandra.apache.org Subject: Re: Cassandra 2.0.x OOM during startsup - schema version inconsistency after reboot Hi Michaels :-), My guess is this ticket will be closed with a "Won't Fix" resolution. Cassandra 2.0 is no longer supported and I have seen tickets being rejected like CASSANDRA-10510<https://issues.apache.org/jira/browse/CASSANDRA-10510>. Would you like to upgrade to 2.1.last and see if you still have the issue? About your issue, do you stop your node using a command like the following one? nodetool disablethrift && nodetool disablebinary && sleep 5 && nodetool disablegossip && sleep 10 && nodetool drain && sleep 10 && sudo service cassandra stop or even flushing: nodetool disablethrift && nodetool disablebinary && sleep 5 && nodetool disablegossip && sleep 10 && nodetool flush && nodetool drain && sleep 10 && sudo service cassandra stop Are commitlogs empty when you start cassandra? C*heers, --- Alain Rodriguez - al...@thelastpickle.com<mailto:al...@thelastpickle.com> France The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2016-05-11 5:35 GMT+02:00 Michael Fong <michael.f...@ruckuswireless.com<mailto:michael.f...@ruckuswireless.com>>: Hi, Thanks for your recommendation. I also opened a ticket to keep track @ https://issues.apache.org/jira/browse/CASSANDRA-11748 Hope this could brought someone's attention to take a look. Thanks. Sincerely, Michael Fong -Original Message- From: Michael Kjellman [mailto:mkjell...@internalcircle.com<mailto:mkjell...@internalcircle.com>] Sent: Monday, May 09, 2016 11:57 AM To: d...@cassandra.apache.org<mailto:d...@cassandra.apache.org> Cc: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Cassandra 2.0.x OOM during startsup - schema version inconsistency after reboot I'd recommend you create a JIRA! That way you can get some traction on the issue. Obviously an OOM is never correct, even if your process is wrong in some way! Best, kjellman Sent from my iPhone > On May 8, 2016, at 8:48 PM, Michael Fong > <michael.f...@ruckuswireless.com<mailto:michael.f...@ruckuswireless.com>> > wrote: > > Hi, all, > > > Haven't heard any responses so far, and this isue has troubled us for quite > some time. Here is another update: > > We have noticed several times that The schema version may change after > migration and reboot: > > Here is the scenario: > > 1. Two node cluster (1 & 2). > > 2. There are some schema changes, i.e. create a few new columnfamily. > The cluster will wait until both nodes have schema version in sync (describe > cluster) before moving on. > > 3. Right before node2 is rebooted, the schema version is consistent; > however, after ndoe2 reboots and starts servicing, the MigrationManager would > gossip different schema version. > > 4. Afterwards, both nodes starts exchanging schema message > indefinitely until one of the node dies. > > We currently suspect the change of schema is due to replying the old entry in > commit log. We wish to continue dig further, but need experts help on this. > > I don't know if anyone has seen this before, or if there is anything wrong > with our migration flow though.. > > Thanks in advance. > > Best regards, > > > Michael Fong > > From: Michael Fong > [mailto:michael.f...@ruckuswireless.com<mailto:michael.f...@ruckuswireless.com>] > Sent: Thursday, April 21, 2016 6:41 PM > To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>; > d...@cassandra.apache.org<mailto:d...@cassandra.apache.org> > Subject: RE: Cassandra 2.0.x OOM during bootstrap > > Hi,
RE: Cassandra 2.0.x OOM during startsup - schema version inconsistency after reboot
Hi, Thanks for your recommendation. I also opened a ticket to keep track @ https://issues.apache.org/jira/browse/CASSANDRA-11748 Hope this could brought someone's attention to take a look. Thanks. Sincerely, Michael Fong -Original Message- From: Michael Kjellman [mailto:mkjell...@internalcircle.com] Sent: Monday, May 09, 2016 11:57 AM To: d...@cassandra.apache.org Cc: user@cassandra.apache.org Subject: Re: Cassandra 2.0.x OOM during startsup - schema version inconsistency after reboot I'd recommend you create a JIRA! That way you can get some traction on the issue. Obviously an OOM is never correct, even if your process is wrong in some way! Best, kjellman Sent from my iPhone > On May 8, 2016, at 8:48 PM, Michael Fong <michael.f...@ruckuswireless.com> > wrote: > > Hi, all, > > > Haven't heard any responses so far, and this isue has troubled us for quite > some time. Here is another update: > > We have noticed several times that The schema version may change after > migration and reboot: > > Here is the scenario: > > 1. Two node cluster (1 & 2). > > 2. There are some schema changes, i.e. create a few new columnfamily. > The cluster will wait until both nodes have schema version in sync (describe > cluster) before moving on. > > 3. Right before node2 is rebooted, the schema version is consistent; > however, after ndoe2 reboots and starts servicing, the MigrationManager would > gossip different schema version. > > 4. Afterwards, both nodes starts exchanging schema message > indefinitely until one of the node dies. > > We currently suspect the change of schema is due to replying the old entry in > commit log. We wish to continue dig further, but need experts help on this. > > I don't know if anyone has seen this before, or if there is anything wrong > with our migration flow though.. > > Thanks in advance. > > Best regards, > > > Michael Fong > > From: Michael Fong [mailto:michael.f...@ruckuswireless.com] > Sent: Thursday, April 21, 2016 6:41 PM > To: user@cassandra.apache.org; d...@cassandra.apache.org > Subject: RE: Cassandra 2.0.x OOM during bootstrap > > Hi, all, > > Here is some more information on before the OOM happened on the rebooted node > in a 2-node test cluster: > > > 1. It seems the schema version has changed on the rebooted node after > reboot, i.e. > Before reboot, > Node 1: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,326 > MigrationManager.java (line 328) Gossiping my schema version > 4cb463f8-5376-3baf-8e88-a5cc6a94f58f > Node 2: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,122 > MigrationManager.java (line 328) Gossiping my schema version > 4cb463f8-5376-3baf-8e88-a5cc6a94f58f > > After rebooting node 2, > Node 2: DEBUG [main] 2016-04-19 11:18:18,016 MigrationManager.java > (line 328) Gossiping my schema version > f5270873-ba1f-39c7-ab2e-a86db868b09b > > > > 2. After reboot, both nods repeatedly send MigrationTask to each other > - we suspect it is related to the schema version (Digest) mismatch after Node > 2 rebooted: > The node2 keeps submitting the migration task over 100+ times to the other > node. > INFO [GossipStage:1] 2016-04-19 11:18:18,261 Gossiper.java (line 1011) > Node /192.168.88.33 has restarted, now UP INFO [GossipStage:1] > 2016-04-19 11:18:18,262 TokenMetadata.java (line 414) Updating > topology for /192.168.88.33 INFO [GossipStage:1] 2016-04-19 > 11:18:18,263 StorageService.java (line 1544) Node /192.168.88.33 state > jump to normal INFO [GossipStage:1] 2016-04-19 11:18:18,264 > TokenMetadata.java (line 414) Updating topology for /192.168.88.33 DEBUG > [GossipStage:1] 2016-04-19 11:18:18,265 MigrationManager.java (line 102) > Submitting migration task for /192.168.88.33 DEBUG [GossipStage:1] 2016-04-19 > 11:18:18,265 MigrationManager.java (line 102) Submitting migration task for > /192.168.88.33 DEBUG [MigrationStage:1] 2016-04-19 11:18:18,268 > MigrationTask.java (line 62) Can't send schema pull request: node > /192.168.88.33 is down. > DEBUG [MigrationStage:1] 2016-04-19 11:18:18,268 MigrationTask.java (line 62) > Can't send schema pull request: node /192.168.88.33 is down. > DEBUG [RequestResponseStage:1] 2016-04-19 11:18:18,353 Gossiper.java > (line 977) removing expire time for endpoint : /192.168.88.33 INFO > [RequestResponseStage:1] 2016-04-19 11:18:18,353 Gossiper.java (line > 978) InetAddress /192.168.88.33 is now UP DEBUG > [RequestResponseStage:1] 2016-04-19 11:18:18,353 MigrationManager.java > (line 102) Submitting migration task for /192.168.88.33 DEBUG > [RequestResponseStage:1] 2016-04-19 11:18:18,355 Gossiper.java (line >
RE: Effectiveness of Scrub Operation vs SSTable previously marked in blacklist
Hi,
I have filed a jira ticket to keep tracked @
https://issues.apache.org/jira/browse/CASSANDRA-11624
Thanks!
Sincerely,
Michael Fong
From: Marcus Eriksson [mailto:krum...@gmail.com]
Sent: Wednesday, March 23, 2016 10:47 PM
To: user@cassandra.apache.org
Subject: Re: Effectiveness of Scrub Operation vs SSTable previously marked in
blacklist
yeah that is most likely a bug, could you file a ticket?
On Tue, Mar 22, 2016 at 4:36 AM, Michael Fong
<michael.f...@ruckuswireless.com<mailto:michael.f...@ruckuswireless.com>> wrote:
Hi, all,
We recently encountered a scenario under Cassandra 2.0 deployment. Cassandra
detected a corrupted sstable, and when we attempt to scrub the sstable (with
all the associated sstables), the corrupted sstable was not included in the
sstable list. This continues until we restart Cassandra and perform sstable
again.
After we traced the Cassandra source code, we are a bit confused with the
effectiveness of scrubbing and SStable being marked in blacklist in Cassandra
2.0+
It seems from previous version (Cassandra 1.2), the scrub operation would
operate on a sstable regardless of it being previously marked. However, in
Cassandra 2.0, the function flows seems changed.
Here is function flow that we traced in Cassandra 2.0 source code:
From org.apache.cassandra.db.compaction.CompactionManager
…
public void performScrub(ColumnFamilyStore cfStore, final boolean
skipCorrupted, final boolean checkData) throws InterruptedException,
ExecutionException
{
performAllSSTableOperation(cfStore, new AllSSTablesOperation()
{
…
private void performAllSSTableOperation(final ColumnFamilyStore cfs, final
AllSSTablesOperation operation) throws InterruptedException, ExecutionException
{
final Iterable sstables = cfs.markAllCompacting();
…
org.apache.cassandra.db. ColumnFamilyStore
…
public Iterable markAllCompacting()
{
Callable<Iterable> callable = new
Callable<Iterable>()
{
public Iterable call() throws Exception
{
assert data.getCompacting().isEmpty() : data.getCompacting();
Iterable sstables =
Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables()));
if (Iterables.isEmpty(sstables))
return null;
…
If it is true, would this flow – marking corrupted sstable in blacklist, defeat
the original purpose of scrub operation? Thanks in advanced!
Sincerely,
Michael Fong
RE: Cassandra 2.0.x OOM during startsup - schema version inconsistency after reboot
Hi, all, Haven't heard any responses so far, and this isue has troubled us for quite some time. Here is another update: We have noticed several times that The schema version may change after migration and reboot: Here is the scenario: 1. Two node cluster (1 & 2). 2. There are some schema changes, i.e. create a few new columnfamily. The cluster will wait until both nodes have schema version in sync (describe cluster) before moving on. 3. Right before node2 is rebooted, the schema version is consistent; however, after ndoe2 reboots and starts servicing, the MigrationManager would gossip different schema version. 4. Afterwards, both nodes starts exchanging schema message indefinitely until one of the node dies. We currently suspect the change of schema is due to replying the old entry in commit log. We wish to continue dig further, but need experts help on this. I don't know if anyone has seen this before, or if there is anything wrong with our migration flow though.. Thanks in advance. Best regards, Michael Fong From: Michael Fong [mailto:michael.f...@ruckuswireless.com] Sent: Thursday, April 21, 2016 6:41 PM To: user@cassandra.apache.org; d...@cassandra.apache.org Subject: RE: Cassandra 2.0.x OOM during bootstrap Hi, all, Here is some more information on before the OOM happened on the rebooted node in a 2-node test cluster: 1. It seems the schema version has changed on the rebooted node after reboot, i.e. Before reboot, Node 1: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,326 MigrationManager.java (line 328) Gossiping my schema version 4cb463f8-5376-3baf-8e88-a5cc6a94f58f Node 2: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,122 MigrationManager.java (line 328) Gossiping my schema version 4cb463f8-5376-3baf-8e88-a5cc6a94f58f After rebooting node 2, Node 2: DEBUG [main] 2016-04-19 11:18:18,016 MigrationManager.java (line 328) Gossiping my schema version f5270873-ba1f-39c7-ab2e-a86db868b09b 2. After reboot, both nods repeatedly send MigrationTask to each other - we suspect it is related to the schema version (Digest) mismatch after Node 2 rebooted: The node2 keeps submitting the migration task over 100+ times to the other node. INFO [GossipStage:1] 2016-04-19 11:18:18,261 Gossiper.java (line 1011) Node /192.168.88.33 has restarted, now UP INFO [GossipStage:1] 2016-04-19 11:18:18,262 TokenMetadata.java (line 414) Updating topology for /192.168.88.33 INFO [GossipStage:1] 2016-04-19 11:18:18,263 StorageService.java (line 1544) Node /192.168.88.33 state jump to normal INFO [GossipStage:1] 2016-04-19 11:18:18,264 TokenMetadata.java (line 414) Updating topology for /192.168.88.33 DEBUG [GossipStage:1] 2016-04-19 11:18:18,265 MigrationManager.java (line 102) Submitting migration task for /192.168.88.33 DEBUG [GossipStage:1] 2016-04-19 11:18:18,265 MigrationManager.java (line 102) Submitting migration task for /192.168.88.33 DEBUG [MigrationStage:1] 2016-04-19 11:18:18,268 MigrationTask.java (line 62) Can't send schema pull request: node /192.168.88.33 is down. DEBUG [MigrationStage:1] 2016-04-19 11:18:18,268 MigrationTask.java (line 62) Can't send schema pull request: node /192.168.88.33 is down. DEBUG [RequestResponseStage:1] 2016-04-19 11:18:18,353 Gossiper.java (line 977) removing expire time for endpoint : /192.168.88.33 INFO [RequestResponseStage:1] 2016-04-19 11:18:18,353 Gossiper.java (line 978) InetAddress /192.168.88.33 is now UP DEBUG [RequestResponseStage:1] 2016-04-19 11:18:18,353 MigrationManager.java (line 102) Submitting migration task for /192.168.88.33 DEBUG [RequestResponseStage:1] 2016-04-19 11:18:18,355 Gossiper.java (line 977) removing expire time for endpoint : /192.168.88.33 INFO [RequestResponseStage:1] 2016-04-19 11:18:18,355 Gossiper.java (line 978) InetAddress /192.168.88.33 is now UP DEBUG [RequestResponseStage:1] 2016-04-19 11:18:18,355 MigrationManager.java (line 102) Submitting migration task for /192.168.88.33 DEBUG [RequestResponseStage:2] 2016-04-19 11:18:18,355 Gossiper.java (line 977) removing expire time for endpoint : /192.168.88.33 INFO [RequestResponseStage:2] 2016-04-19 11:18:18,355 Gossiper.java (line 978) InetAddress /192.168.88.33 is now UP DEBUG [RequestResponseStage:2] 2016-04-19 11:18:18,356 MigrationManager.java (line 102) Submitting migration task for /192.168.88.33 . On the otherhand, Node 1 keeps updating its gossip information, followed by receiving and submitting migrationTask afterwards: DEBUG [RequestResponseStage:3] 2016-04-19 11:18:18,332 Gossiper.java (line 977) removing expire time for endpoint : /192.168.88.34 INFO [RequestResponseStage:3] 2016-04-19 11:18:18,333 Gossiper.java (line 978) InetAddress /192.168.88.34 is now UP DEBUG [RequestResponseStage:4] 2016-04-19 11:18:18,335 Gossiper.java (line 977) removing expire time for endpoint : /192.168.88.34 INFO [RequestResponseStage:4] 2016-04-19 11:18:18,335 Gossiper.java (line 978) InetAdd
RE: Cassandra 2.0.x OOM during bootstrap
in advanced! Sincerely, Michael Fong From: Michael Fong [mailto:michael.f...@ruckuswireless.com] Sent: Wednesday, April 20, 2016 10:43 AM To: user@cassandra.apache.org; d...@cassandra.apache.org Subject: Cassandra 2.0.x OOM during bootstrap Hi, all, We have recently encountered a Cassandra OOM issue when Cassandra is brought up sometimes (but not always) in our 4-node cluster test bed. After analyzing the heap dump, we could find the Internal-Response thread pool (JMXEnabledThreadPoolExecutor) is filled with thounds of 'org.apache.cassandra.net.MessageIn' objects, and occupy > 2 gigabytes of heap memory. According to the documents on internet, it seems internal-response thread pool is somewhat related to schema-checking. Has anyone encountered similar issue before? We are using Cassandra 2.0.17 and JDK 1.8. Thanks in advance! Sincerely, Michael Fong
Cassandra 2.0.x OOM during bootstrap
Hi, all, We have recently encountered a Cassandra OOM issue when Cassandra is brought up sometimes (but not always) in our 4-node cluster test bed. After analyzing the heap dump, we could find the Internal-Response thread pool (JMXEnabledThreadPoolExecutor) is filled with thounds of 'org.apache.cassandra.net.MessageIn' objects, and occupy > 2 gigabytes of heap memory. According to the documents on internet, it seems internal-response thread pool is somewhat related to schema-checking. Has anyone encountered similar issue before? We are using Cassandra 2.0.17 and JDK 1.8. Thanks in advance! Sincerely, Michael Fong
RE: C* 1.2.x vs Gossip marking DOWN/UP
Hi, Alain,
Thanks for your reply.
Unfortunately, it is a rather old version of system which comes with Cassandra
v1.2.15, and database upgrade does not seem to be a viable solution. We have
also recently observed a situation that the Cassandra instance froze around one
minute while the other nodes eventually mark that node DOWN. Here are some logs
of the scenario, that there is a 1 minute window with no sign of any operation
was running:
Gossip related :
TRACE [GossipStage:1] 2016-04-13 23:34:08,641 GossipDigestSynVerbHandler.java
(line 40) Received a GossipDigestSynMessage from /156.1.1.1
TRACE [GossipStage:1] 2016-04-13 23:35:01,081 GossipDigestSynVerbHandler.java
(line 71) Gossip syn digests are : /156.1.1.1:1460103192:520418
/156.1.1.4:1460103190:522108 /156.1.1.2:1460103205:522912
/156.1.1.3:1460551526:41979
GC related:
2016-04-13T23:34:02.675+: 487270.189: Total time for which application
threads were stopped: 0.0677060 seconds
2016-04-13T23:35:01.019+: 487328.533: [GC2016-04-13T23:35:01.020+:
487328.534: [ParNew
Desired survivor size 1474560 bytes, new threshold 1 (max 1)
- age 1:1637144 bytes,1637144 total
: 843200K->1600K(843200K), 0.0559840 secs] 5631683K->4814397K(8446400K),
0.0567850 secs] [Times: user=0.67 sys=0.00, real=0.05 secs]
Regular Cassandra operation:
INFO [CompactionExecutor:70229] 2016-04-13 23:34:02,439 CompactionTask.java
(line 266) Compacted 4 sstables to
[/opt/ruckuswireless/wsg/db/data/wsg/indexHistoricalRuckusClient/wsg-indexHistoricalRuckusClient-ic-1464,].
54,743,298 bytes to 53,661,608 (~98% of original) in 29,124ms = 1.757166MB/s.
417,517 total rows, 265,853 unique. Row merge counts were {1:114862,
2:150328, 3:653, 4:10, }
INFO [HANDSHAKE-/156.1.1.2] 2016-04-13 23:35:01,110 OutboundTcpConnection.java
(line 418) Handshaking version with /156.1.1.2
The situation comes randomly among all nodes. When this happens, the hector
client application seems to have trouble connecting to that Cassandra database
as well, for example,
04-13 23:34:54 [taskExecutor-167] ConcurrentHClientPool:273 ERROR - Transport
exception in re-opening client in release on
:{localhost(127.0.0.1):9160}
Has anyone had similar experience? The operating system is Ubuntu and kernel
version is 2.6.32.24. Thanks in advance!
Sincerely,
Michael fong
From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
Sent: Wednesday, April 13, 2016 9:30 PM
To: user@cassandra.apache.org
Subject: Re: C* 1.2.x vs Gossip marking DOWN/UP
Hi Michael,
I had critical issues using 1.2 (.11, I believe) around gossip (but it was like
2 years ago...).
Are you using the last C* 1.2.19 minor version? If not, you probably should go
there asap.
A lot of issues like this one
https://issues.apache.org/jira/browse/CASSANDRA-6297 have been fixed since then
on C* 1.2, 2.0, 2.1, 2.2, 3.0.X, 3.X. You got to go through steps to upgrade.
It should be safe and enough to go to the last 1.2 minor to solve this issue.
For your information, even C* 2.0 is no longer supported. The minimum version
you should use now is 2.1.last.
This technical debt might end up costing you more in terms of time, money and
Quality of Service that taking care of upgrades. The most probable thing is
that your bug is fixed already on newer versions. Plus it is not very
interesting for us to help you as we would have to go through old code, to find
issues that are most likely already fixed. If you want some support (from
community or commercial one) you really should upgrade this cluster. Make sure
your clients are compatible too.
I did not know that some people were still using C* < 2.0 :-).
Cheers,
---
Alain Rodriguez - al...@thelastpickle.com<mailto:al...@thelastpickle.com>
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-04-13 10:58 GMT+02:00 Michael Fong
<michael.f...@ruckuswireless.com<mailto:michael.f...@ruckuswireless.com>>:
Hi, all
We have been a Cassandra 4-node cluster (C* 1.2.x) where a node marked all the
other 3 nodes DOWN, and came back UP a few seconds later. There was a
compaction that kicked in a minute before, roughly 10~MB in size, followed by
marking all the other nodes DOWN later. In the other words, in the system.log
we see
00:00:00 Compacting ….
00:00:03 Compacted 8 sstables … 10~ megabytes
00:01:06 InetAddress /x.x.x.4 is now DOWN
00:01:06 InetAddress /x.x.x.3 is now DOWN
00:01:06 InetAddress /x.x.x.1 is now DOWN
There was no significant GC activities in gc.log. We have heard that busy
compaction activities would cause this behavior, but we cannot reason why this
could happen logically. How come a compaction operation would stop the Gossip
thread to perform heartbeat check? Has anyone experienced this kind of behavior
before?
Thanks in advanced!
Sincerely,
Michael Fong
C* 1.2.x vs Gossip marking DOWN/UP
Hi, all We have been a Cassandra 4-node cluster (C* 1.2.x) where a node marked all the other 3 nodes DOWN, and came back UP a few seconds later. There was a compaction that kicked in a minute before, roughly 10~MB in size, followed by marking all the other nodes DOWN later. In the other words, in the system.log we see 00:00:00 Compacting 00:00:03 Compacted 8 sstables ... 10~ megabytes 00:01:06 InetAddress /x.x.x.4 is now DOWN 00:01:06 InetAddress /x.x.x.3 is now DOWN 00:01:06 InetAddress /x.x.x.1 is now DOWN There was no significant GC activities in gc.log. We have heard that busy compaction activities would cause this behavior, but we cannot reason why this could happen logically. How come a compaction operation would stop the Gossip thread to perform heartbeat check? Has anyone experienced this kind of behavior before? Thanks in advanced! Sincerely, Michael Fong
RE: Gossip heartbeat and packet capture
Hi, Thanks for your comment. The Cassandra version is in 1.2.15, and we have also adjusted the phi_convict_threshold to 12 in production. This setting works great in most of the production cases, except for this particular one... Also, adding more node is not a plausible option for now. :( After going through the source code of C* 1.2.15, the serialized gossip heartbeat message (SYN/ACK/ACK2) seem to contain 1. Cluster name, 2. Partitioner name in the payload. Perhaps I could grep the gossip heartbeat packet by filtering this criteria? Sincerely, Michael Fong From: sean_r_dur...@homedepot.com [mailto:sean_r_dur...@homedepot.com] Sent: Wednesday, March 23, 2016 10:39 PM To: user@cassandra.apache.org Subject: RE: Gossip heartbeat and packet capture Is this from the 1.1 line, perhaps? In my experience it could be very flappy for no particular reason we could discover. 1.1 is a pretty dusty version. Upgrading into the 2.1 or later would be a good idea. If you have to upgrade in place without down time, you will need to go through many upgrades to get to the latest versions. (The late versions of 1.2 are pretty stable, though.) Also, you can look at the phi_convict_threshold Cassandra.yaml parameter to help with network flakiness. Default is 8, I believe in most versions. Increasing to 10 or 12 might help. Finally, a 2 node cluster doesn't give you many of the benefits of a distributed system. Consider adding some nodes, if you can. Sean Durity - Lead Cassandra Admin Big DATA Team For support, create a JIRA<https://portal.homedepot.com/sites/bigdata/Shared%20Documents/Jira%20Hadoop%20Support%20Workflow.pdf> From: Michael Fong [mailto:michael.f...@ruckuswireless.com] Sent: Wednesday, March 23, 2016 4:12 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Gossip heartbeat and packet capture Hi, all, We are trying to reason the possible scenarios when a C*(v1.x) cluster connection keeps flapping in production. (Two node cluster, each node keeps marking the other node DOWN but came back UP within seconds; multiple times a day) We have checked the load on the cluster i- very light and low GC activities also. We have also checked the network interface / devices were working just fine on the nodes during the incidence. We are changing our investigation direction to the network topology/settings, so we are thinking to capture gossip heartbeat packet to verify if the packet is received as expected on the other end. Has anyone tried to capture the packet of gossip internode communication? What would be the filter / criteria to grep heartbeat-related packet only? Thanks in advance! Michael The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Gossip heartbeat and packet capture
Hi, all, We are trying to reason the possible scenarios when a C*(v1.x) cluster connection keeps flapping in production. (Two node cluster, each node keeps marking the other node DOWN but came back UP within seconds; multiple times a day) We have checked the load on the cluster i- very light and low GC activities also. We have also checked the network interface / devices were working just fine on the nodes during the incidence. We are changing our investigation direction to the network topology/settings, so we are thinking to capture gossip heartbeat packet to verify if the packet is received as expected on the other end. Has anyone tried to capture the packet of gossip internode communication? What would be the filter / criteria to grep heartbeat-related packet only? Thanks in advance! Michael
Effectiveness of Scrub Operation vs SSTable previously marked in blacklist
Hi, all,
We recently encountered a scenario under Cassandra 2.0 deployment. Cassandra
detected a corrupted sstable, and when we attempt to scrub the sstable (with
all the associated sstables), the corrupted sstable was not included in the
sstable list. This continues until we restart Cassandra and perform sstable
again.
After we traced the Cassandra source code, we are a bit confused with the
effectiveness of scrubbing and SStable being marked in blacklist in Cassandra
2.0+
It seems from previous version (Cassandra 1.2), the scrub operation would
operate on a sstable regardless of it being previously marked. However, in
Cassandra 2.0, the function flows seems changed.
Here is function flow that we traced in Cassandra 2.0 source code:
>From org.apache.cassandra.db.compaction.CompactionManager
...
public void performScrub(ColumnFamilyStore cfStore, final boolean
skipCorrupted, final boolean checkData) throws InterruptedException,
ExecutionException
{
performAllSSTableOperation(cfStore, new AllSSTablesOperation()
{
...
private void performAllSSTableOperation(final ColumnFamilyStore cfs, final
AllSSTablesOperation operation) throws InterruptedException, ExecutionException
{
final Iterable sstables = cfs.markAllCompacting();
...
org.apache.cassandra.db. ColumnFamilyStore
...
public Iterable markAllCompacting()
{
Callable<Iterable> callable = new
Callable<Iterable>()
{
public Iterable call() throws Exception
{
assert data.getCompacting().isEmpty() : data.getCompacting();
Iterable sstables =
Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables()));
if (Iterables.isEmpty(sstables))
return null;
...
If it is true, would this flow - marking corrupted sstable in blacklist, defeat
the original purpose of scrub operation? Thanks in advanced!
Sincerely,
Michael Fong
