Re: Index creation sometimes fails
Hi Tyler, FWIW I was not able to reproduce this problem with a smaller example. I'll go ahead and file the JIRA anyway. Thanks for your help! Best regards, Clint On Thu, Jul 17, 2014 at 3:05 PM, Tyler Hobbs ty...@datastax.com wrote: On Thu, Jul 17, 2014 at 4:59 PM, Clint Kelly clint.ke...@gmail.com wrote: I will post a JIRA, along with directions on how to get this to happen. The tricky thing, though, is that this doesn't always happen, and I cannot reproduce it on my laptop or in a VM. Even if you can't reproduce, just include as many details as you can. C* and driver versions, schemas, logs, etc. BTW you mean the datastax JIRA, correct? Oops! Yes, I meant https://datastax-oss.atlassian.net/browse/JAVA, not the batch statement docs :) -- Tyler Hobbs DataStax http://datastax.com/
Re: Index creation sometimes fails
Hi Tyler, Thanks for replying. This is good to know that I am not going crazy! :) I will post a JIRA, along with directions on how to get this to happen. The tricky thing, though, is that this doesn't always happen, and I cannot reproduce it on my laptop or in a VM. BTW you mean the datastax JIRA, correct? Best regards, Clint On Wed, Jul 16, 2014 at 4:32 PM, Tyler Hobbs ty...@datastax.com wrote: On Tue, Jul 15, 2014 at 1:40 PM, Clint Kelly clint.ke...@gmail.com wrote: Is there some way to get the driver to block until the schema code has propagated everywhere? My currently solution feels rather janky! The driver *should* be blocking until the schema has propagated already. If it's not, that's a bug. I would check the changelog and JIRA for related tickets, and if you don't find anything, open a new ticket with details and steps to repro: http://cassandra.apache.org/doc/cql3/CQL.html#batchStmt -- Tyler Hobbs DataStax
Re: Index creation sometimes fails
On Tue, Jul 15, 2014 at 1:40 PM, Clint Kelly clint.ke...@gmail.com wrote: Is there some way to get the driver to block until the schema code has propagated everywhere? My currently solution feels rather janky! The driver *should* be blocking until the schema has propagated already. If it's not, that's a bug. I would check the changelog and JIRA for related tickets, and if you don't find anything, open a new ticket with details and steps to repro: http://cassandra.apache.org/doc/cql3/CQL.html#batchStmt -- Tyler Hobbs DataStax http://datastax.com/
Re: Index creation sometimes fails
FWIW I was able to work around this problem by having my code run the following loop: while (the index doesn't exit and we haven't hit our limit of number of tries): try to create the index read from system.schema_columns to see if the index exists if so, then break if not, pause 5 seconds This loop took three iterations to create the index. Is this expected? This seems really weird! Best regards, Clint On Mon, Jul 14, 2014 at 5:54 PM, Clint Kelly clint.ke...@gmail.com wrote: BTW I have seen this using versions 2.0.1 and 2.0.3 of the java driver on a three-node cluster with DSE 4.5. On Mon, Jul 14, 2014 at 5:51 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, I have some code that I've been fiddling with today that uses the DataStax Java driver to create a table and then create a secondary index on a column in that table. I've testing this code fairly thoroughly on a single-node Cassandra instance on my laptop and in unit test (using the CassandraDaemon). When running on a three-node cluster, however, I see strange behavior. Although my table always gets created, the secondary index often does not! If I delete the table and then create it again (through the same code that I've written), I've never seen the index fail to appear the second time. Does anyone have any idea what to look for here? I have no experience working on a Cassandra cluster and I wonder if maybe I am doing something dumb (I basically just installed DSE and started up the three nodes and that was it). I don't see anything that looks unusual in OpsCenter for DSE. The only thing I've noticed is that the presence of output like the following from my program after executing the command to create the index is perfectly correlated with successful creation of the index: 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Received event EVENT CREATED kiji_retail2.t_model_repo, scheduling delivery 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: [Control connection] Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [b309518a-35d2-3790-bb66-ea39bb0d188c] If anyone can give me a hand, I would really appreciate it. I am out of ideas! Best regards, Clint
Re: Index creation sometimes fails
As far as I know, schema propagation always takes some times in the cluster. On this mailing list some people in the past faced similar behavior. On Tue, Jul 15, 2014 at 8:20 PM, Clint Kelly clint.ke...@gmail.com wrote: FWIW I was able to work around this problem by having my code run the following loop: while (the index doesn't exit and we haven't hit our limit of number of tries): try to create the index read from system.schema_columns to see if the index exists if so, then break if not, pause 5 seconds This loop took three iterations to create the index. Is this expected? This seems really weird! Best regards, Clint On Mon, Jul 14, 2014 at 5:54 PM, Clint Kelly clint.ke...@gmail.com wrote: BTW I have seen this using versions 2.0.1 and 2.0.3 of the java driver on a three-node cluster with DSE 4.5. On Mon, Jul 14, 2014 at 5:51 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, I have some code that I've been fiddling with today that uses the DataStax Java driver to create a table and then create a secondary index on a column in that table. I've testing this code fairly thoroughly on a single-node Cassandra instance on my laptop and in unit test (using the CassandraDaemon). When running on a three-node cluster, however, I see strange behavior. Although my table always gets created, the secondary index often does not! If I delete the table and then create it again (through the same code that I've written), I've never seen the index fail to appear the second time. Does anyone have any idea what to look for here? I have no experience working on a Cassandra cluster and I wonder if maybe I am doing something dumb (I basically just installed DSE and started up the three nodes and that was it). I don't see anything that looks unusual in OpsCenter for DSE. The only thing I've noticed is that the presence of output like the following from my program after executing the command to create the index is perfectly correlated with successful creation of the index: 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Received event EVENT CREATED kiji_retail2.t_model_repo, scheduling delivery 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: [Control connection] Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [b309518a-35d2-3790-bb66-ea39bb0d188c] If anyone can give me a hand, I would really appreciate it. I am out of ideas! Best regards, Clint
Re: Index creation sometimes fails
Hi DuyHai, Thanks for the response. Is the recommended best practice therefore to do what I describe above? Is there some way to get the driver to block until the schema code has propagated everywhere? My currently solution feels rather janky! Thanks! Best regards, Clint On Tue, Jul 15, 2014 at 11:32 AM, DuyHai Doan doanduy...@gmail.com wrote: As far as I know, schema propagation always takes some times in the cluster. On this mailing list some people in the past faced similar behavior. On Tue, Jul 15, 2014 at 8:20 PM, Clint Kelly clint.ke...@gmail.com wrote: FWIW I was able to work around this problem by having my code run the following loop: while (the index doesn't exit and we haven't hit our limit of number of tries): try to create the index read from system.schema_columns to see if the index exists if so, then break if not, pause 5 seconds This loop took three iterations to create the index. Is this expected? This seems really weird! Best regards, Clint On Mon, Jul 14, 2014 at 5:54 PM, Clint Kelly clint.ke...@gmail.com wrote: BTW I have seen this using versions 2.0.1 and 2.0.3 of the java driver on a three-node cluster with DSE 4.5. On Mon, Jul 14, 2014 at 5:51 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, I have some code that I've been fiddling with today that uses the DataStax Java driver to create a table and then create a secondary index on a column in that table. I've testing this code fairly thoroughly on a single-node Cassandra instance on my laptop and in unit test (using the CassandraDaemon). When running on a three-node cluster, however, I see strange behavior. Although my table always gets created, the secondary index often does not! If I delete the table and then create it again (through the same code that I've written), I've never seen the index fail to appear the second time. Does anyone have any idea what to look for here? I have no experience working on a Cassandra cluster and I wonder if maybe I am doing something dumb (I basically just installed DSE and started up the three nodes and that was it). I don't see anything that looks unusual in OpsCenter for DSE. The only thing I've noticed is that the presence of output like the following from my program after executing the command to create the index is perfectly correlated with successful creation of the index: 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Received event EVENT CREATED kiji_retail2.t_model_repo, scheduling delivery 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: [Control connection] Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [b309518a-35d2-3790-bb66-ea39bb0d188c] If anyone can give me a hand, I would really appreciate it. I am out of ideas! Best regards, Clint
Index creation sometimes fails
Hi everyone, I have some code that I've been fiddling with today that uses the DataStax Java driver to create a table and then create a secondary index on a column in that table. I've testing this code fairly thoroughly on a single-node Cassandra instance on my laptop and in unit test (using the CassandraDaemon). When running on a three-node cluster, however, I see strange behavior. Although my table always gets created, the secondary index often does not! If I delete the table and then create it again (through the same code that I've written), I've never seen the index fail to appear the second time. Does anyone have any idea what to look for here? I have no experience working on a Cassandra cluster and I wonder if maybe I am doing something dumb (I basically just installed DSE and started up the three nodes and that was it). I don't see anything that looks unusual in OpsCenter for DSE. The only thing I've noticed is that the presence of output like the following from my program after executing the command to create the index is perfectly correlated with successful creation of the index: 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Received event EVENT CREATED kiji_retail2.t_model_repo, scheduling delivery 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: [Control connection] Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [b309518a-35d2-3790-bb66-ea39bb0d188c] If anyone can give me a hand, I would really appreciate it. I am out of ideas! Best regards, Clint
Re: Index creation sometimes fails
BTW I have seen this using versions 2.0.1 and 2.0.3 of the java driver on a three-node cluster with DSE 4.5. On Mon, Jul 14, 2014 at 5:51 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, I have some code that I've been fiddling with today that uses the DataStax Java driver to create a table and then create a secondary index on a column in that table. I've testing this code fairly thoroughly on a single-node Cassandra instance on my laptop and in unit test (using the CassandraDaemon). When running on a three-node cluster, however, I see strange behavior. Although my table always gets created, the secondary index often does not! If I delete the table and then create it again (through the same code that I've written), I've never seen the index fail to appear the second time. Does anyone have any idea what to look for here? I have no experience working on a Cassandra cluster and I wonder if maybe I am doing something dumb (I basically just installed DSE and started up the three nodes and that was it). I don't see anything that looks unusual in OpsCenter for DSE. The only thing I've noticed is that the presence of output like the following from my program after executing the command to create the index is perfectly correlated with successful creation of the index: 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Received event EVENT CREATED kiji_retail2.t_model_repo, scheduling delivery 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: [Control connection] Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [b309518a-35d2-3790-bb66-ea39bb0d188c] If anyone can give me a hand, I would really appreciate it. I am out of ideas! Best regards, Clint
