Hey Michael, In the case that you have a production cluster set up with multiple nodes, assuming you have rf>1 it’s easier to just replace the broken node and restore it’s data. (For future reference)
I wasn’t sure if view was referring to materialised view at the time although Pradeeps comment along with your own suggest it might (I didn’t get a chance to look through the code to confirm if view was MV or something else and I’m not that familiar with the code base). As far as the choice of using Materialised Views, they aren’t being deprecated they are currently marked as experimental and most people strongly advise you to not use them. If you can avoid it don’t do it. They’re associated with a lot of bugs and scalability issues. Also they’re just hard to do right if you aren’t exceptionally familiar with Cassandra. Regards, Evelyn. > On 7 Jun 2018, at 3:05 am, Pradeep Chhetri <prad...@stashaway.com> wrote: > > Hi Michael, > > We have faced the same situation as yours in our production environment where > we suddenly got "Unknown CF Exception" for materialized views too. We are > using Lagom apps with cassandra for persistence. In our case, since these > views can be regenerated from the original events, we were able to safely > recover. > > Few suggestions from my operations experience: > > 1) Upgrade your cassandra cluster to 3.11.2 because there are lots of bug > fixes specific to materialized views. > 2) Never let your application create/update/delete cassandra > table/materialized views. Always create them manually to make sure that only > connection is doing the operation. > > Regards, > Pradeep > > > > On Wed, Jun 6, 2018 at 9:44 PM, <m...@vis.at <mailto:m...@vis.at>> wrote: > Hi Evelyn, > > thanks a lot for your detailed response message. > > The data is not important. We've already wiped the data and created a new > cassandra installation. The data re-import task is already running. We've > lost the data for a couple of months but in this case this does not matter. > > Nevertheless we will try what you told us - just to be smarter/faster if this > happens in production (where we will setup a cassandra cluster with multiple > cassandra nodes anyway). I will drop you a note when we are done. > > Hmmm... the problem is within a "View". Are this the materialized views? > > I'm asking this because: > * Someone on the internet (stackoverflow if a recall correctly) mentioned > that using materialized views are to be deprecated. > * I had been on a datastax workshop in Zurich a couple of days ago where a > datastax employee told me that we should not use materialized views - it is > better to create & fill all tables directly. > > Would you also recommend not to use materialized views? As this problem is > related to a view - maybe we could avoid this problem simply by following > this recommendation. > > Thanks a lot again! > > Greetings, > Michael > > > > > On 06.06.2018 16:48, Evelyn Smith wrote: > Hi Michael, > > So I looked at the code, here are some stages of your error message: > 1. at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292) > [apache-cassandra-3.11.0.jar:3.11.0 > At this step Cassandra is running through the keyspaces in it’s > schema turning off compactions for all tables before it starts > rerunning the commit log (so it isn’t an issue with the commit log). > 2. at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127) > ~[apache-cassandra-3.11.0.jar:3.11.0] > Loading key space related to the column family that is erroring out > 3. at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324) > ~[apache-cassandra-3.11.0.jar:3.11.0] > Cassandra has initialised the column family and is reloading the view > 4. at > org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204) > ~[apache-cassandra-3.11.0.jar:3.11.0] > At this point I haven’t had enough time to tell if Cassandra is > requesting info on a column specifically or still requesting > information on a column family. Regardless, given we already rule out > issues with the SSTables and their directory and Cassandra is yet to > start processing the commit log this to me suggests it’s something > wrong in one of the system keyspaces storing the schema information. > > There should definitely be a way to resolve this with zero data loss > by either: > 1. Fixing the issue in the system keyspace SSTables (hard) > 2. Rerunning the commit log on a new Cassandra node that has been > restored from the current one (I’m not sure if this is possible but > I’ll figure it out tomorrow) > > The alternative is if you are ok with losing the commitlog then you > can backup the data and restore it to a new node (or the same node but > with everything blown away). This isn’t a trivial process though > I’ve done it a few times. > > How important is the data? > > Happy to come back to this tomorrow (need some sleep) > > Regards, > Eevee. > > On 5 Jun 2018, at 7:32 pm, m...@vis.at <mailto:m...@vis.at> wrote: > Keyspace.getColumnFamilyStore > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > <mailto:user-unsubscr...@cassandra.apache.org> > For additional commands, e-mail: user-h...@cassandra.apache.org > <mailto:user-h...@cassandra.apache.org> > >
