[
https://issues.apache.org/jira/browse/CASSANDRA-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
NIcholas Rushton updated CASSANDRA-13133:
-----------------------------------------
Description:
Started to notice many open file descriptors (100k+) per node, growing at a
rate of about 30 per minute in our cluster. After turning off our JMX exporting
server(https://github.com/prometheus/jmx_exporter), which gets queried every 30
seconds, the number of file descriptors remained static.
Digging a bit further I ran a jmx dump tool over all the cassandra metrics and
tracked the number of file descriptors after each query, boiling it down to a
single metric causing the number of file descriptors to increase:
org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
running a query a few times against this metric shows the file descriptors
increasing after each query:
{code}
for _ in {0..3}
do
java -jar jmx-dump-0.4.2-standalone.jar --port 7199 --dump
org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> /dev/null;
sudo lsof -p `pgrep -f CassandraDaemon` | fgrep "DIR" | awk
'{a[$(NF)]+=1}END{for(k in a){print k, a[k]}}' | grep "events_by"
done
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> 33176
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> 33177
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> 33178
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> 33179
{code}
it should be noted that the file descriptor is open on a directory, not an
actual file
was:
Started to notice many open file descriptors (100k+) per node, growing at a
rate of about 30 per minute in our cluster. After turning off our JMX exporting
server which gets queried every 30 seconds, the number of file descriptors
remained static.
Digging a bit further I ran a jmx dump tool over all the cassandra metrics and
tracked the number of file descriptors after each query, boiling it down to a
single metric causing the number of file descriptors to increase:
org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
running a query a few times against this metric shows the file descriptors
increasing after each query:
{code}
for _ in {0..3}
do
java -jar jmx-dump-0.4.2-standalone.jar --port 7199 --dump
org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> /dev/null;
sudo lsof -p `pgrep -f CassandraDaemon` | fgrep "DIR" | awk
'{a[$(NF)]+=1}END{for(k in a){print k, a[k]}}' | grep "events_by"
done
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> 33176
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> 33177
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> 33178
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> 33179
{code}
it should be noted that the file descriptor is open on a directory, not an
actual file
> Unclosed file descriptors when querying SnapshotsSize metric
> ------------------------------------------------------------
>
> Key: CASSANDRA-13133
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13133
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: CentOS 7
> Reporter: NIcholas Rushton
> Labels: jmx, lhf, metrics, newbie
> Fix For: 3.9
>
>
> Started to notice many open file descriptors (100k+) per node, growing at a
> rate of about 30 per minute in our cluster. After turning off our JMX
> exporting server(https://github.com/prometheus/jmx_exporter), which gets
> queried every 30 seconds, the number of file descriptors remained static.
> Digging a bit further I ran a jmx dump tool over all the cassandra metrics
> and tracked the number of file descriptors after each query, boiling it down
> to a single metric causing the number of file descriptors to increase:
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> running a query a few times against this metric shows the file descriptors
> increasing after each query:
> {code}
> for _ in {0..3}
> do
> java -jar jmx-dump-0.4.2-standalone.jar --port 7199 --dump
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> > /dev/null;
> sudo lsof -p `pgrep -f CassandraDaemon` | fgrep "DIR" | awk
> '{a[$(NF)]+=1}END{for(k in a){print k, a[k]}}' | grep "events_by"
> done
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> > 33176
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> > 33177
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> > 33178
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> > 33179
> {code}
> it should be noted that the file descriptor is open on a directory, not an
> actual file
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
