Re: Running enablefullquerylog crashes cassandra
Are there really no entries after those INFO messages? That indicates that a person/script/daemon/tool/process killed Cassandra. Perhaps check the OS logs to see if oom-killer kicked in to see if the C* process was terminated. Cheers! >
Re: Running enablefullquerylog crashes cassandra
Nothing in the cassandra logs (system/debug) last lines are INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,334 BinLog.java:420 - Attempting to configure bin log: Path: /mnt/fql_data Roll cycle: HOURLY Blocking: true Max queue weight: 268435456 Max log size:34359738368 Archive command: INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,335 BinLog.java:433 - Cleaning directory: /mnt/fql_data as requested On Sun, Feb 6, 2022 at 6:43 PM Jeff Jirsa wrote: > That looks like a nodetool stack - can you check the Cassandra log for > corresponding error? > > On Feb 6, 2022, at 12:52 AM, Gil Ganz wrote: > > > Hey > I'm trying to enable full query log on cassandra 4.01 node and it's > causing cassandra to shutdown > > nodetool enablefullquerylog --path /mnt/fql_data > > Cassandra has shutdown. > error: null > -- StackTrace -- > java.io.EOFException > at java.io.DataInputStream.readByte(DataInputStream.java:267) > at > sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222) > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) > at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) > at > javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) > at > javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) > at > javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) > at com.sun.proxy.$Proxy6.enableFullQueryLogger(Unknown Source) > at > org.apache.cassandra.tools.NodeProbe.enableFullQueryLogger(NodeProbe.java:1836) > at > org.apache.cassandra.tools.nodetool.EnableFullQueryLog.execute(EnableFullQueryLog.java:62) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:358) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:343) > at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:246) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:84) > > /mnt/fql_data is owned by cassandra user > Doesn't matter if directory is empty or not > > contents of cassandra.yaml > > full_query_logging_options: > log_dir: /mnt/fql_data > roll_cycle: HOURLY > block: true > max_queue_weight: 268435456 > max_log_size: 34359738368 > # archive command is "/path/to/script.sh %path" where %path is > replaced with the file being rolled: > # archive_command: > # max_archive_retries: 10 > > No errors in the log, last couple of lines are > > INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,334 > BinLog.java:420 - Attempting to configure bin log: Path: /mnt/fql_data Roll > cycle: HOURLY Blocking: true Max queue weight: 268435456 Max log > size:34359738368 Archive command: > INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,335 > BinLog.java:433 - Cleaning directory: /mnt/fql_data as requested > > I noticed there is a similiar bug > https://issues.apache.org/jira/browse/CASSANDRA-17136 but I also tried > setting disk_failure_policy to ignore, same thing. > Has Anyone encountered something similar? > > > > > > gil > >
Re: Running enablefullquerylog crashes cassandra
That looks like a nodetool stack - can you check the Cassandra log for corresponding error? > On Feb 6, 2022, at 12:52 AM, Gil Ganz wrote: > > > Hey > I'm trying to enable full query log on cassandra 4.01 node and it's causing > cassandra to shutdown > > nodetool enablefullquerylog --path /mnt/fql_data > > Cassandra has shutdown. > error: null > -- StackTrace -- > java.io.EOFException > at java.io.DataInputStream.readByte(DataInputStream.java:267) > at > sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222) > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) > at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) > at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown > Source) > at > javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) > at > javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) > at com.sun.proxy.$Proxy6.enableFullQueryLogger(Unknown Source) > at > org.apache.cassandra.tools.NodeProbe.enableFullQueryLogger(NodeProbe.java:1836) > at > org.apache.cassandra.tools.nodetool.EnableFullQueryLog.execute(EnableFullQueryLog.java:62) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:358) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:343) > at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:246) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:84) > > /mnt/fql_data is owned by cassandra user > Doesn't matter if directory is empty or not > > contents of cassandra.yaml > > full_query_logging_options: > log_dir: /mnt/fql_data > roll_cycle: HOURLY > block: true > max_queue_weight: 268435456 > max_log_size: 34359738368 > # archive command is "/path/to/script.sh %path" where %path is replaced > with the file being rolled: > # archive_command: > # max_archive_retries: 10 > > No errors in the log, last couple of lines are > > INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,334 > BinLog.java:420 - Attempting to configure bin log: Path: /mnt/fql_data Roll > cycle: HOURLY Blocking: true Max queue weight: 268435456 Max log > size:34359738368 Archive command: > INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,335 > BinLog.java:433 - Cleaning directory: /mnt/fql_data as requested > > I noticed there is a similiar bug > https://issues.apache.org/jira/browse/CASSANDRA-17136 but I also tried > setting disk_failure_policy to ignore, same thing. > Has Anyone encountered something similar? > > > > > > gil
Running enablefullquerylog crashes cassandra
Hey I'm trying to enable full query log on cassandra 4.01 node and it's causing cassandra to shutdown nodetool enablefullquerylog --path /mnt/fql_data Cassandra has shutdown. error: null -- StackTrace -- java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:267) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) at com.sun.proxy.$Proxy6.enableFullQueryLogger(Unknown Source) at org.apache.cassandra.tools.NodeProbe.enableFullQueryLogger(NodeProbe.java:1836) at org.apache.cassandra.tools.nodetool.EnableFullQueryLog.execute(EnableFullQueryLog.java:62) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:358) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:343) at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:246) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:84) /mnt/fql_data is owned by cassandra user Doesn't matter if directory is empty or not contents of cassandra.yaml full_query_logging_options: log_dir: /mnt/fql_data roll_cycle: HOURLY block: true max_queue_weight: 268435456 max_log_size: 34359738368 # archive command is "/path/to/script.sh %path" where %path is replaced with the file being rolled: # archive_command: # max_archive_retries: 10 No errors in the log, last couple of lines are INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,334 BinLog.java:420 - Attempting to configure bin log: Path: /mnt/fql_data Roll cycle: HOURLY Blocking: true Max queue weight: 268435456 Max log size:34359738368 Archive command: INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,335 BinLog.java:433 - Cleaning directory: /mnt/fql_data as requested I noticed there is a similiar bug https://issues.apache.org/jira/browse/CASSANDRA-17136 but I also tried setting disk_failure_policy to ignore, same thing. Has Anyone encountered something similar? gil
