Re: Written data is lost and no exception thrown back to the client
I have the same problem. When I bulk load my data, I have a problem with Cassandra Datastax driver. dependency groupIdcom.datastax.cassandra/groupId artifactIdcassandra-driver-core/artifactId version2.1.4/version !-- Driver 2.1.6, 2.1.7.1 gives problems. Some data is lost. -- /dependency With version 2.1.6 and also with version 2.1.7.1 I have lost records with no error message what so ever. With version 2.1.4 I have no missing records. I use CL.ONE to write my records. I use RF 3. On 21 Aug 2015, at 13:06 , Robert Wille rwi...@fold3.commailto:rwi...@fold3.com wrote: But it shouldn’t matter. I have missing data, and no errors, which shouldn’t be possible except with CL=ANY. FWIW, I’m working on some sample code so I can post a Jira. Robert On Aug 21, 2015, at 5:04 AM, Robert Wille rwi...@fold3.commailto:rwi...@fold3.com wrote: RF=1 with QUORUM consistency. I know QUORUM is weird with RF=1, but it should be the same as ONE. If’s QUORUM instead of ONE because production has RF=3, and I was running this against my test cluster with RF=1. On Aug 20, 2015, at 7:28 PM, Jason jkushm...@rocketfuelinc.commailto:jkushm...@rocketfuelinc.com wrote: What consistency level were the writes? From: Robert Willemailto:rwi...@fold3.com Sent: 8/20/2015 18:25 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Written data is lost and no exception thrown back to the client I wrote a data migration application which I was testing, and I pushed it too hard and the FlushWriter thread pool blocked, and I ended up with dropped mutation messages. I compared the source data against what is in my cluster, and as expected I have missing records. The strange thing is that my application didn’t error out. I’ve been doing some forensics, and there’s a lot about this that makes no sense and makes me feel very uneasy. I use a lot of asynchronous queries, and I thought it was possible that I had bad error handling, so I checked for errors in other, independent ways. I have a retry policy that on the first failure logs the error and then requests a retry. On the second failure it logs the error and then rethrows. A few retryable errors appeared in my logs, but no fatal errors. In theory, I should have a fatal error in my logs for any error that gets reported back to the client. I wrap my Session object, and all queries go through this wrapper. This wrapper logs all query errors. Synchronous queries are wrapped in a try/catch which logs and rethrows. Asynchronous queries use a FutureCallback to log any onFailure invocations. My logs indicate that no errors whatsoever were reported back to me. I do not understand how I can get dropped mutation messages and not know about it. I am running 2.0.16 with datastax Java driver 2.0.8. Three node cluster with RF=1. If someone could help me understand how this can occur, I would greatly appreciate it. A database that errors out is one thing. A database that errors out and makes you think everything was fine is quite another. Thanks Robert
Re: Written data is lost and no exception thrown back to the client
RF=1 with QUORUM consistency. I know QUORUM is weird with RF=1, but it should be the same as ONE. If’s QUORUM instead of ONE because production has RF=3, and I was running this against my test cluster with RF=1. On Aug 20, 2015, at 7:28 PM, Jason jkushm...@rocketfuelinc.commailto:jkushm...@rocketfuelinc.com wrote: What consistency level were the writes? From: Robert Willemailto:rwi...@fold3.com Sent: 8/20/2015 18:25 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Written data is lost and no exception thrown back to the client I wrote a data migration application which I was testing, and I pushed it too hard and the FlushWriter thread pool blocked, and I ended up with dropped mutation messages. I compared the source data against what is in my cluster, and as expected I have missing records. The strange thing is that my application didn’t error out. I’ve been doing some forensics, and there’s a lot about this that makes no sense and makes me feel very uneasy. I use a lot of asynchronous queries, and I thought it was possible that I had bad error handling, so I checked for errors in other, independent ways. I have a retry policy that on the first failure logs the error and then requests a retry. On the second failure it logs the error and then rethrows. A few retryable errors appeared in my logs, but no fatal errors. In theory, I should have a fatal error in my logs for any error that gets reported back to the client. I wrap my Session object, and all queries go through this wrapper. This wrapper logs all query errors. Synchronous queries are wrapped in a try/catch which logs and rethrows. Asynchronous queries use a FutureCallback to log any onFailure invocations. My logs indicate that no errors whatsoever were reported back to me. I do not understand how I can get dropped mutation messages and not know about it. I am running 2.0.16 with datastax Java driver 2.0.8. Three node cluster with RF=1. If someone could help me understand how this can occur, I would greatly appreciate it. A database that errors out is one thing. A database that errors out and makes you think everything was fine is quite another. Thanks Robert
Re: Written data is lost and no exception thrown back to the client
But it shouldn’t matter. I have missing data, and no errors, which shouldn’t be possible except with CL=ANY. FWIW, I’m working on some sample code so I can post a Jira. Robert On Aug 21, 2015, at 5:04 AM, Robert Wille rwi...@fold3.commailto:rwi...@fold3.com wrote: RF=1 with QUORUM consistency. I know QUORUM is weird with RF=1, but it should be the same as ONE. If’s QUORUM instead of ONE because production has RF=3, and I was running this against my test cluster with RF=1. On Aug 20, 2015, at 7:28 PM, Jason jkushm...@rocketfuelinc.commailto:jkushm...@rocketfuelinc.com wrote: What consistency level were the writes? From: Robert Willemailto:rwi...@fold3.com Sent: 8/20/2015 18:25 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Written data is lost and no exception thrown back to the client I wrote a data migration application which I was testing, and I pushed it too hard and the FlushWriter thread pool blocked, and I ended up with dropped mutation messages. I compared the source data against what is in my cluster, and as expected I have missing records. The strange thing is that my application didn’t error out. I’ve been doing some forensics, and there’s a lot about this that makes no sense and makes me feel very uneasy. I use a lot of asynchronous queries, and I thought it was possible that I had bad error handling, so I checked for errors in other, independent ways. I have a retry policy that on the first failure logs the error and then requests a retry. On the second failure it logs the error and then rethrows. A few retryable errors appeared in my logs, but no fatal errors. In theory, I should have a fatal error in my logs for any error that gets reported back to the client. I wrap my Session object, and all queries go through this wrapper. This wrapper logs all query errors. Synchronous queries are wrapped in a try/catch which logs and rethrows. Asynchronous queries use a FutureCallback to log any onFailure invocations. My logs indicate that no errors whatsoever were reported back to me. I do not understand how I can get dropped mutation messages and not know about it. I am running 2.0.16 with datastax Java driver 2.0.8. Three node cluster with RF=1. If someone could help me understand how this can occur, I would greatly appreciate it. A database that errors out is one thing. A database that errors out and makes you think everything was fine is quite another. Thanks Robert
Re: Written data is lost and no exception thrown back to the client
Please let us know the Jira number. All the best, [image: datastax_logo.png] http://www.datastax.com/ Sebastián Estévez Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com [image: linkedin.png] https://www.linkedin.com/company/datastax [image: facebook.png] https://www.facebook.com/datastax [image: twitter.png] https://twitter.com/datastax [image: g+.png] https://plus.google.com/+Datastax/about http://feeds.feedburner.com/datastax http://cassandrasummit-datastax.com/?utm_campaign=summit15utm_medium=summiticonutm_source=emailsignature DataStax is the fastest, most scalable distributed database technology, delivering Apache Cassandra to the world’s most innovative enterprises. Datastax is built to be agile, always-on, and predictably scalable to any size. With more than 500 customers in 45 countries, DataStax is the database technology and transactional backbone of choice for the worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay. On Fri, Aug 21, 2015 at 7:06 AM, Robert Wille rwi...@fold3.com wrote: But it shouldn’t matter. I have missing data, and no errors, which shouldn’t be possible except with CL=ANY. FWIW, I’m working on some sample code so I can post a Jira. Robert On Aug 21, 2015, at 5:04 AM, Robert Wille rwi...@fold3.com wrote: RF=1 with QUORUM consistency. I know QUORUM is weird with RF=1, but it should be the same as ONE. If’s QUORUM instead of ONE because production has RF=3, and I was running this against my test cluster with RF=1. On Aug 20, 2015, at 7:28 PM, Jason jkushm...@rocketfuelinc.com wrote: What consistency level were the writes? -- From: Robert Wille rwi...@fold3.com Sent: 8/20/2015 18:25 To: user@cassandra.apache.org Subject: Written data is lost and no exception thrown back to the client I wrote a data migration application which I was testing, and I pushed it too hard and the FlushWriter thread pool blocked, and I ended up with dropped mutation messages. I compared the source data against what is in my cluster, and as expected I have missing records. The strange thing is that my application didn’t error out. I’ve been doing some forensics, and there’s a lot about this that makes no sense and makes me feel very uneasy. I use a lot of asynchronous queries, and I thought it was possible that I had bad error handling, so I checked for errors in other, independent ways. I have a retry policy that on the first failure logs the error and then requests a retry. On the second failure it logs the error and then rethrows. A few retryable errors appeared in my logs, but no fatal errors. In theory, I should have a fatal error in my logs for any error that gets reported back to the client. I wrap my Session object, and all queries go through this wrapper. This wrapper logs all query errors. Synchronous queries are wrapped in a try/catch which logs and rethrows. Asynchronous queries use a FutureCallback to log any onFailure invocations. My logs indicate that no errors whatsoever were reported back to me. I do not understand how I can get dropped mutation messages and not know about it. I am running 2.0.16 with datastax Java driver 2.0.8. Three node cluster with RF=1. If someone could help me understand how this can occur, I would greatly appreciate it. A database that errors out is one thing. A database that errors out and makes you think everything was fine is quite another. Thanks Robert
RE: Written data is lost and no exception thrown back to the client
What consistency level were the writes? -Original Message- From: Robert Wille rwi...@fold3.com Sent: 8/20/2015 18:25 To: user@cassandra.apache.org user@cassandra.apache.org Subject: Written data is lost and no exception thrown back to the client I wrote a data migration application which I was testing, and I pushed it too hard and the FlushWriter thread pool blocked, and I ended up with dropped mutation messages. I compared the source data against what is in my cluster, and as expected I have missing records. The strange thing is that my application didn’t error out. I’ve been doing some forensics, and there’s a lot about this that makes no sense and makes me feel very uneasy. I use a lot of asynchronous queries, and I thought it was possible that I had bad error handling, so I checked for errors in other, independent ways. I have a retry policy that on the first failure logs the error and then requests a retry. On the second failure it logs the error and then rethrows. A few retryable errors appeared in my logs, but no fatal errors. In theory, I should have a fatal error in my logs for any error that gets reported back to the client. I wrap my Session object, and all queries go through this wrapper. This wrapper logs all query errors. Synchronous queries are wrapped in a try/catch which logs and rethrows. Asynchronous queries use a FutureCallback to log any onFailure invocations. My logs indicate that no errors whatsoever were reported back to me. I do not understand how I can get dropped mutation messages and not know about it. I am running 2.0.16 with datastax Java driver 2.0.8. Three node cluster with RF=1. If someone could help me understand how this can occur, I would greatly appreciate it. A database that errors out is one thing. A database that errors out and makes you think everything was fine is quite another. Thanks Robert
Written data is lost and no exception thrown back to the client
I wrote a data migration application which I was testing, and I pushed it too hard and the FlushWriter thread pool blocked, and I ended up with dropped mutation messages. I compared the source data against what is in my cluster, and as expected I have missing records. The strange thing is that my application didn’t error out. I’ve been doing some forensics, and there’s a lot about this that makes no sense and makes me feel very uneasy. I use a lot of asynchronous queries, and I thought it was possible that I had bad error handling, so I checked for errors in other, independent ways. I have a retry policy that on the first failure logs the error and then requests a retry. On the second failure it logs the error and then rethrows. A few retryable errors appeared in my logs, but no fatal errors. In theory, I should have a fatal error in my logs for any error that gets reported back to the client. I wrap my Session object, and all queries go through this wrapper. This wrapper logs all query errors. Synchronous queries are wrapped in a try/catch which logs and rethrows. Asynchronous queries use a FutureCallback to log any onFailure invocations. My logs indicate that no errors whatsoever were reported back to me. I do not understand how I can get dropped mutation messages and not know about it. I am running 2.0.16 with datastax Java driver 2.0.8. Three node cluster with RF=1. If someone could help me understand how this can occur, I would greatly appreciate it. A database that errors out is one thing. A database that errors out and makes you think everything was fine is quite another. Thanks Robert
