[jira] [Commented] (CASSANDRA-12813) NPE in auth for bootstrapping node
[ https://issues.apache.org/jira/browse/CASSANDRA-12813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150908#comment-16150908 ] Andres March commented on CASSANDRA-12813: -- any workaround? I got this on a 3.9 cluster with a new node bootstrapping. No upgrades from another version. > NPE in auth for bootstrapping node > -- > > Key: CASSANDRA-12813 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12813 > Project: Cassandra > Issue Type: Bug >Reporter: Charles Mims >Assignee: Alex Petrov > Fix For: 2.2.9, 3.0.10, 3.10 > > > {code} > ERROR [SharedPool-Worker-1] 2016-10-19 21:40:25,991 Message.java:617 - > Unexpected exception during request; channel = [id: 0x15eb017f, / omitted>:40869 => /10.0.0.254:9042] > java.lang.NullPointerException: null > at > org.apache.cassandra.auth.PasswordAuthenticator.doAuthenticate(PasswordAuthenticator.java:144) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:86) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.auth.PasswordAuthenticator.access$100(PasswordAuthenticator.java:54) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.auth.PasswordAuthenticator$PlainTextSaslAuthenticator.getAuthenticatedUser(PasswordAuthenticator.java:182) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:78) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513) > [apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407) > [apache-cassandra-3.0.9.jar:3.0.9] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_101] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > [apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.9.jar:3.0.9] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > {code} > I have a node that has been joining for around 24 hours. My application is > configured with the IP address of the joining node in the list of nodes to > connect to (ruby driver), and I have been getting around 200 events of this > NPE per hour. I removed the IP of the joining node from the list of nodes > for my app to connect to and the errors stopped. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13334) Cassandra fails to start with LOCAL_JMX set to false
[ https://issues.apache.org/jira/browse/CASSANDRA-13334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059900#comment-16059900 ] Andres March commented on CASSANDRA-13334: -- i totally understand. this ticket is just asking for an error message instead of silent failure > Cassandra fails to start with LOCAL_JMX set to false > > > Key: CASSANDRA-13334 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13334 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: cassandra 3.0.9 > ubuntu 16.04 > ec2 >Reporter: Andres March > Fix For: 3.11.x > > > There seems to be some changes lately with CASSANDRA-11540 and > CASSANDRA-11725 that changes how remote jmx set up works. > Just setting LOCAL_JMX=false causes cassandra to fail to start on ubuntu in > ec2. Worse yet the initd script says it has started ok and there are no logs > whatsoever about an error. > This is reproducible 100% of the time. Changing various settings like the > port, jmx remote hostname, etc... didn't change anything. I would attach > some logs but there aren't any. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12924) GraphiteReporter does not reconnect if graphite restarts
[ https://issues.apache.org/jira/browse/CASSANDRA-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andres March updated CASSANDRA-12924: - Status: Patch Available (was: Open) https://github.com/apache/cassandra/pull/123 > GraphiteReporter does not reconnect if graphite restarts > > > Key: CASSANDRA-12924 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12924 > Project: Cassandra > Issue Type: Bug >Reporter: Stefano Ortolani > > Seems like GraphiteReporter does not reconnect after graphite is restarted. > The consequence is complete loss of reported metrics until Cassandra > restarts. Logs show this every minute: > {noformat} > WARN [metrics-graphite-reporter-1-thread-1] 2016-11-17 10:06:26,549 > GraphiteReporter.java:179 - Unable to report to Graphite > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > ~[na:1.8.0_91] > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) > ~[na:1.8.0_91] > at java.net.SocketOutputStream.write(SocketOutputStream.java:153) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) ~[na:1.8.0_91] > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) > ~[na:1.8.0_91] > at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129) > ~[na:1.8.0_91] > at java.io.BufferedWriter.write(BufferedWriter.java:230) ~[na:1.8.0_91] > at java.io.Writer.write(Writer.java:157) ~[na:1.8.0_91] > at com.codahale.metrics.graphite.Graphite.send(Graphite.java:130) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:283) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) > [metrics-core-3.1.0.jar:3.1.0] > at > com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) > [metrics-core-3.1.0.jar:3.1.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_91] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_91] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_91] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_91] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_91] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_91] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > WARN [metrics-graphite-reporter-1-thread-1] 2016-11-17 10:06:26,549 > GraphiteReporter.java:183 - Error closing Graphite > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > ~[na:1.8.0_91] > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) > ~[na:1.8.0_91] > at java.net.SocketOutputStream.write(SocketOutputStream.java:153) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) ~[na:1.8.0_91] > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) > ~[na:1.8.0_91] > at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129) > ~[na:1.8.0_91] > at java.io.BufferedWriter.write(BufferedWriter.java:230) ~[na:1.8.0_91] > at java.io.Writer.write(Writer.java:157) ~[na:1.8.0_91] > at com.codahale.metrics.graphite.Graphite.send(Graphite.java:130) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:283) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) > [metrics-core-3.1.0.jar:3.1.0] > at > com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) > [metrics-core-3.1.0.jar:3.1.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.jav
[jira] [Commented] (CASSANDRA-12924) GraphiteReporter does not reconnect if graphite restarts
[ https://issues.apache.org/jira/browse/CASSANDRA-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056133#comment-16056133 ] Andres March commented on CASSANDRA-12924: -- https://github.com/apache/cassandra/pull/123 This would be really nice to include in 3.11 metrics-core update is required. > GraphiteReporter does not reconnect if graphite restarts > > > Key: CASSANDRA-12924 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12924 > Project: Cassandra > Issue Type: Bug >Reporter: Stefano Ortolani > > Seems like GraphiteReporter does not reconnect after graphite is restarted. > The consequence is complete loss of reported metrics until Cassandra > restarts. Logs show this every minute: > {noformat} > WARN [metrics-graphite-reporter-1-thread-1] 2016-11-17 10:06:26,549 > GraphiteReporter.java:179 - Unable to report to Graphite > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > ~[na:1.8.0_91] > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) > ~[na:1.8.0_91] > at java.net.SocketOutputStream.write(SocketOutputStream.java:153) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) ~[na:1.8.0_91] > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) > ~[na:1.8.0_91] > at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129) > ~[na:1.8.0_91] > at java.io.BufferedWriter.write(BufferedWriter.java:230) ~[na:1.8.0_91] > at java.io.Writer.write(Writer.java:157) ~[na:1.8.0_91] > at com.codahale.metrics.graphite.Graphite.send(Graphite.java:130) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:283) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) > [metrics-core-3.1.0.jar:3.1.0] > at > com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) > [metrics-core-3.1.0.jar:3.1.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_91] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_91] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_91] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_91] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_91] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_91] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > WARN [metrics-graphite-reporter-1-thread-1] 2016-11-17 10:06:26,549 > GraphiteReporter.java:183 - Error closing Graphite > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > ~[na:1.8.0_91] > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) > ~[na:1.8.0_91] > at java.net.SocketOutputStream.write(SocketOutputStream.java:153) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) > ~[na:1.8.0_91] > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) ~[na:1.8.0_91] > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) > ~[na:1.8.0_91] > at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129) > ~[na:1.8.0_91] > at java.io.BufferedWriter.write(BufferedWriter.java:230) ~[na:1.8.0_91] > at java.io.Writer.write(Writer.java:157) ~[na:1.8.0_91] > at com.codahale.metrics.graphite.Graphite.send(Graphite.java:130) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:283) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158) > ~[metrics-graphite-3.1.0.jar:3.1.0] > at > com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) > [metrics-core-3.1.0.jar:3.1.0] > at > com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) > [metrics-core-
[jira] [Commented] (CASSANDRA-13334) Cassandra fails to start with LOCAL_JMX set to false
[ https://issues.apache.org/jira/browse/CASSANDRA-13334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021830#comment-16021830 ] Andres March commented on CASSANDRA-13334: -- The issue isn't that it doesn't start if there are other required settings. It's that it doesn't print a useful error. When I changed the logging to debug on everything, I was able to trace the error. Unfortunately, that was so long ago that I don't remember the cause. If I revisit, I'll note the error and submit a patch. The init script should fail with an error logged. > Cassandra fails to start with LOCAL_JMX set to false > > > Key: CASSANDRA-13334 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13334 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: cassandra 3.0.9 > ubuntu 16.04 > ec2 >Reporter: Andres March > Fix For: 3.11.0 > > > There seems to be some changes lately with CASSANDRA-11540 and > CASSANDRA-11725 that changes how remote jmx set up works. > Just setting LOCAL_JMX=false causes cassandra to fail to start on ubuntu in > ec2. Worse yet the initd script says it has started ok and there are no logs > whatsoever about an error. > This is reproducible 100% of the time. Changing various settings like the > port, jmx remote hostname, etc... didn't change anything. I would attach > some logs but there aren't any. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13334) Cassandra fails to start with LOCAL_JMX set to false
[ https://issues.apache.org/jira/browse/CASSANDRA-13334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andres March updated CASSANDRA-13334: - Reproduced In: 3.9 (was: 3.0.9) Fix Version/s: (was: 3.0.x) 3.11.0 > Cassandra fails to start with LOCAL_JMX set to false > > > Key: CASSANDRA-13334 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13334 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: cassandra 3.0.9 > ubuntu 16.04 > ec2 >Reporter: Andres March > Fix For: 3.11.0 > > > There seems to be some changes lately with CASSANDRA-11540 and > CASSANDRA-11725 that changes how remote jmx set up works. > Just setting LOCAL_JMX=false causes cassandra to fail to start on ubuntu in > ec2. Worse yet the initd script says it has started ok and there are no logs > whatsoever about an error. > This is reproducible 100% of the time. Changing various settings like the > port, jmx remote hostname, etc... didn't change anything. I would attach > some logs but there aren't any. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13334) Cassandra fails to start with LOCAL_JMX set to false
Andres March created CASSANDRA-13334: Summary: Cassandra fails to start with LOCAL_JMX set to false Key: CASSANDRA-13334 URL: https://issues.apache.org/jira/browse/CASSANDRA-13334 Project: Cassandra Issue Type: Bug Components: Lifecycle Environment: cassandra 3.0.9 ubuntu 16.04 ec2 Reporter: Andres March Fix For: 3.0.x There seems to be some changes lately with CASSANDRA-11540 and CASSANDRA-11725 that changes how remote jmx set up works. Just setting LOCAL_JMX=false causes cassandra to fail to start on ubuntu in ec2. Worse yet the initd script says it has started ok and there are no logs whatsoever about an error. This is reproducible 100% of the time. Changing various settings like the port, jmx remote hostname, etc... didn't change anything. I would attach some logs but there aren't any. -- This message was sent by Atlassian JIRA (v6.3.15#6346)