Re: nodetool repair exception
The official recommendation is 100k: http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html I wonder if there's an advantage to this over unlimited if you're running servers which are dedicated to your Cassandra cluster (which you should be for anything production). On Fri Dec 05 2014 at 2:39:24 PM Robert Coli rc...@eventbrite.com wrote: On Wed, Dec 3, 2014 at 6:37 AM, Rafał Furmański rfurman...@opera.com wrote: I see “Too many open files” exception in logs, but I’m sure that my limit is now 150k. Should I increase it? What’s the reasonable limit of open files for cassandra? Why provide any limit? ulimit allows unlimited? =Rob
Re: nodetool repair exception
On Sat, Dec 6, 2014 at 8:05 AM, Eric Stevens migh...@gmail.com wrote: The official recommendation is 100k: http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html I wonder if there's an advantage to this over unlimited if you're running servers which are dedicated to your Cassandra cluster (which you should be for anything production). There is the potential to have monitoring systems, and other small agents, running on systems in production. I could see this simply as a stop-gap to prevent Cassandra from being able to starve the system of free file descriptors. In theory, if there's not a proper watchdog on your monitors this could prevent an issue from causing an alert. However, just a potential advantage I could think of. Cheers! -Tim On Fri Dec 05 2014 at 2:39:24 PM Robert Coli rc...@eventbrite.com wrote: On Wed, Dec 3, 2014 at 6:37 AM, Rafał Furmański rfurman...@opera.com wrote: I see “Too many open files” exception in logs, but I’m sure that my limit is now 150k. Should I increase it? What’s the reasonable limit of open files for cassandra? Why provide any limit? ulimit allows unlimited? =Rob
Re: nodetool repair exception
On Wed, Dec 3, 2014 at 6:37 AM, Rafał Furmański rfurman...@opera.com wrote: I see “Too many open files” exception in logs, but I’m sure that my limit is now 150k. Should I increase it? What’s the reasonable limit of open files for cassandra? Why provide any limit? ulimit allows unlimited? =Rob
nodetool repair exception
Hi All! We have a 8 nodes cluster in 2 DC (4 per DC, RF=3) running Cassandra 2.1.2 on Linux Debian Wheezy. I executed “nodetool repair” on one of the nodes, and this command returned following error: Exception occurred during clean-up. java.lang.reflect.UndeclaredThrowableException error: JMX connection closed. You should check server log for repair status of keyspace sync(Subsequent keyspaces are not going to be repaired). -- StackTrace -- java.io.IOException: JMX connection closed. You should check server log for repair status of keyspace sync(Subsequent keyspaces are not going to be repaired). at org.apache.cassandra.tools.RepairRunner.handleNotification(NodeProbe.java:1351) at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:274) at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:339) at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:324) at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:247) at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441) at javax.management.remote.rmi.RMIConnector.access$1100(RMIConnector.java:121) at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1505) at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1350) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:587) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:470) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:451) at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:107) This error was followed by lots of “Lost Notification” messages. Node became unusable and I had to restart it. Is this an issue? Rafal signature.asc Description: Message signed with OpenPGP using GPGMail
Re: nodetool repair exception
As the exception indicates, nodetool just lost communication with the Cassandra node and cannot print progress any further. Check your system.log on the node, and see if your repair was completed. If there is no error, then it should be fine. On Wed, Dec 3, 2014 at 5:08 AM, Rafał Furmański rfurman...@opera.com wrote: Hi All! We have a 8 nodes cluster in 2 DC (4 per DC, RF=3) running Cassandra 2.1.2 on Linux Debian Wheezy. I executed “nodetool repair” on one of the nodes, and this command returned following error: Exception occurred during clean-up. java.lang.reflect.UndeclaredThrowableException error: JMX connection closed. You should check server log for repair status of keyspace sync(Subsequent keyspaces are not going to be repaired). -- StackTrace -- java.io.IOException: JMX connection closed. You should check server log for repair status of keyspace sync(Subsequent keyspaces are not going to be repaired). at org.apache.cassandra.tools.RepairRunner.handleNotification(NodeProbe.java:1351) at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:274) at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:339) at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:324) at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:247) at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441) at javax.management.remote.rmi.RMIConnector.access$1100(RMIConnector.java:121) at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1505) at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1350) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:587) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:470) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:451) at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:107) This error was followed by lots of “Lost Notification” messages. Node became unusable and I had to restart it. Is this an issue? Rafal -- Yuki Morishita t:yukim (http://twitter.com/yukim)
Re: nodetool repair exception
I see “Too many open files” exception in logs, but I’m sure that my limit is now 150k. Should I increase it? What’s the reasonable limit of open files for cassandra? On 3 gru 2014, at 15:02, Yuki Morishita mor.y...@gmail.com wrote: As the exception indicates, nodetool just lost communication with the Cassandra node and cannot print progress any further. Check your system.log on the node, and see if your repair was completed. If there is no error, then it should be fine. On Wed, Dec 3, 2014 at 5:08 AM, Rafał Furmański rfurman...@opera.com wrote: Hi All! We have a 8 nodes cluster in 2 DC (4 per DC, RF=3) running Cassandra 2.1.2 on Linux Debian Wheezy. I executed “nodetool repair” on one of the nodes, and this command returned following error: Exception occurred during clean-up. java.lang.reflect.UndeclaredThrowableException error: JMX connection closed. You should check server log for repair status of keyspace sync(Subsequent keyspaces are not going to be repaired). -- StackTrace -- java.io.IOException: JMX connection closed. You should check server log for repair status of keyspace sync(Subsequent keyspaces are not going to be repaired). at org.apache.cassandra.tools.RepairRunner.handleNotification(NodeProbe.java:1351) at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:274) at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:339) at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:324) at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:247) at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441) at javax.management.remote.rmi.RMIConnector.access$1100(RMIConnector.java:121) at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1505) at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1350) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:587) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:470) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:451) at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:107) This error was followed by lots of “Lost Notification” messages. Node became unusable and I had to restart it. Is this an issue? Rafal -- Yuki Morishita t:yukim (http://twitter.com/yukim) signature.asc Description: Message signed with OpenPGP using GPGMail
