[
https://issues.apache.org/jira/browse/CASSANDRA-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14550493#comment-14550493
]
Sam Tunnicliffe commented on CASSANDRA-9419:
--------------------------------------------
I'm still seeing failures locally, where the log output looks like this (with
some of the WARNs elided):
{noformat}
[junit] Testsuite: org.apache.cassandra.pig.CqlTableTest
[junit] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
47.007 sec
[junit]
[junit] ------------- Standard Output ---------------
...
[junit] WARN 13:27:58 job_local_0003
[junit] java.io.IOException:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (tried: localhost/127.0.0.1:9042
(com.datastax.driver.core.exceptions.DriverException: Connection thread
interrupted))
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:329)
~[main/:na]
[junit] Caused by:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (tried: localhost/127.0.0.1:9042
(com.datastax.driver.core.exceptions.DriverException: Connection thread
interrupted))
[junit] at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1230)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at com.datastax.driver.core.Cluster.init(Cluster.java:157)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at com.datastax.driver.core.Cluster.connect(Cluster.java:245)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:320)
~[main/:na]
[junit] WARN 13:27:58 job_local_0003
[junit] java.io.IOException:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (tried: localhost/127.0.0.1:9042
(com.datastax.driver.core.exceptions.DriverException: Connection thread
interrupted))
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:329)
~[main/:na]
[junit] Caused by:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (tried: localhost/127.0.0.1:9042
(com.datastax.driver.core.exceptions.DriverException: Connection thread
interrupted))
[junit] at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1230)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at com.datastax.driver.core.Cluster.init(Cluster.java:157)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at com.datastax.driver.core.Cluster.connect(Cluster.java:245)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:320)
~[main/:na]
[junit] WARN 13:27:58 Ooops! Some job has failed! Specify -stop_on_failure
if you want Pig to stop immediately on failure.
[junit] WARN 13:27:58 Ooops! Some job has failed! Specify -stop_on_failure
if you want Pig to stop immediately on failure.
[junit] ERROR 13:27:58 1 map reduce job(s) failed!
[junit] ERROR 13:27:58 1 map reduce job(s) failed!
[junit] WARN 13:27:59 Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
[junit] WARN 13:27:59 Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
[junit] WARN 13:27:59 No job jar file set. User classes may not be found.
See JobConf(Class) or JobConf#setJar(String).
[junit] WARN 13:27:59 No job jar file set. User classes may not be found.
See JobConf(Class) or JobConf#setJar(String).
[junit] WARN 13:28:02 Failed to get RunningJob for job job_local_0004
[junit] WARN 13:28:02 Failed to get RunningJob for job job_local_0004
[junit] WARN 13:28:02 SchemaTupleBackend has already been initialized
[junit] WARN 13:28:02 SchemaTupleBackend has already been initialized
[junit] WARN 13:28:03 Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
[junit] WARN 13:28:03 Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
[junit] WARN 13:28:03 No job jar file set. User classes may not be found.
See JobConf(Class) or JobConf#setJar(String).
[junit] WARN 13:28:03 No job jar file set. User classes may not be found.
See JobConf(Class) or JobConf#setJar(String).
[junit] WARN 13:28:03 Output path is null in cleanup
[junit] WARN 13:28:03 Output path is null in cleanup
[junit] WARN 13:28:03 job_local_0005
[junit] java.io.IOException: java.lang.RuntimeException: failed to prepare
cql query update cql3ks.collectiontable set n = ? WHERE "m" = ?
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:368)
~[main/:na]
[junit] Caused by: java.lang.RuntimeException: failed to prepare cql query
update cql3ks.collectiontable set n = ? WHERE "m" = ?
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.preparedStatement(CqlRecordWriter.java:392)
~[main/:na]
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:346)
~[main/:na]
[junit] Caused by:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (no host was tried)
[junit] at
com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:289)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:79)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.preparedStatement(CqlRecordWriter.java:388)
~[main/:na]
[junit] ... 1 common frames omitted
[junit] Caused by:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (no host was tried)
[junit] at
com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:107)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.SessionManager.execute(SessionManager.java:538)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.SessionManager.prepareAsync(SessionManager.java:124)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:77)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] ... 2 common frames omitted
[junit] WARN 13:28:03 job_local_0005
[junit] java.io.IOException: java.lang.RuntimeException: failed to prepare
cql query update cql3ks.collectiontable set n = ? WHERE "m" = ?
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:368)
~[main/:na]
[junit] Caused by: java.lang.RuntimeException: failed to prepare cql query
update cql3ks.collectiontable set n = ? WHERE "m" = ?
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.preparedStatement(CqlRecordWriter.java:392)
~[main/:na]
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:346)
~[main/:na]
[junit] Caused by:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (no host was tried)
[junit] at
com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:289)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:79)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.preparedStatement(CqlRecordWriter.java:388)
~[main/:na]
[junit] ... 1 common frames omitted
[junit] Caused by:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (no host was tried)
[junit] at
com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:107)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.SessionManager.execute(SessionManager.java:538)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.SessionManager.prepareAsync(SessionManager.java:124)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] at
com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:77)
~[cassandra-driver-core-2.1.5-shaded.jar:na]
[junit] ... 2 common frames omitted
{noformat}
I think you want to add a {{continue}} at the end of the catch block where the
connection is opened. Otherwise when the connection attempt fails, unless we
meet the conditions to {{break:outer}}, it goes on to try and prepare the
statement, which results in a {{RuntimeException}}.
> CqlRecordWriter does not adequately determine if driver failed to connect due
> to interrupt
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-9419
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9419
> Project: Cassandra
> Issue Type: Bug
> Reporter: Philip Thompson
> Assignee: Philip Thompson
> Fix For: 2.2.0 rc1
>
> Attachments: 9419.txt
>
>
> Until https://datastax-oss.atlassian.net/browse/JAVA-769 is implemented, I
> have had to implement a workaround.
> Java's thread interrupt status flag can be cleared for any number of reasons.
> After discussion, I've changed to parsing the driver's DriverException to
> tell if we are safe to retry the failed connection.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
