[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. and we found that
many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, found this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task after compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. I think it should wrong using unbound queue. If we using
unbound queue, the thread pool wouldn't increasing thread even there're many
tasks are blocked in queue, because unbound queue never would be full. I think
here should use bound queue, so when clean task is heavily, more threads would
created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. and we found that
many files like
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. and we found that
many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, found this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task after compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. I think it should wrong using unbound queue. If we using
unbound queue, the thread pool wouldn't increasing thread even there so many
task are blocked in queue, because unbound queue never would be full. I think
here should use bound queue, so when task is heavily, more threads would
created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. and we found that
many files like
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. and we found that
many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, found this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task and compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. and we found that
many files like
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. and we found that
many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, found this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task after compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. and we found that
many files like
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. we found that so
many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, we found that this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task and compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstable consume so many
disk space, it's causing no enough disk for saving real data. we found that so
many files
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstable consume so many
disk space, it's causing no enough disk for saving real data. we found that so
many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, we found that this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task and compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so many
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. and we found that
many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, we found that this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task and compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, these old sstables consume so many
disk space, it's causing no enough disk for saving real data. we found that so
many files
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is throwed out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, we found that this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task and compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrower out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrown out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, we found that this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task and compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is throwed out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last , we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrower out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, we found that this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task and compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading. last , we found this problem
is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrower out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrower out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, we found that this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task and compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading data. last , we found this
problem is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrower out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "file map count" is 1
million, these values should enough for loading. last , we found this problem
is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrower out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, we found that this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task and compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }
{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "map count" is 1
million, these values should enough for loading. last , we found this problem
is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrower out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so many files
[jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
[
https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Xie updated CASSANDRA-14653:
--
Description:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met some problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "map count" is 1
million, these values should enough for loading. last , we found this problem
is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrower out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so many files like "mc_txn_compaction_xxx.log" are created under the data
directory.
After some times' investigaton, we found that this problem is caused by
"NonPeriodicTasks" thread pools. this pools is always using only one thread
for processing clean task and compaction. this thread pool is instanced with
class DebuggableScheduledThreadPoolExecutor,
and DebuggableScheduledThreadPoolExecutor is inherit from class
ScheduledThreadPoolExecutor.
By reading the code of class DebuggableScheduledThreadPoolExecutor, found
DebuggableScheduledThreadPoolExecutor is using an unbound task queue, and core
pool size is 1. Why here use the unbound queue for queuing submitted tasks?
If we using unbound queue, the thread pool wouldn't increasing thread even
there so many task are blocked in queue, because unbound queue never would be
full. I think here should use bound queue, so when task is heavily, more
threads would created for processing them.
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String
threadPoolName, int priority)
Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName,
priority)); setRejectedExecutionHandler(rejectedExecutionHandler); }
public ScheduledThreadPoolExecutor(int corePoolSize,
ThreadFactory threadFactory)
Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new
DelayedWorkQueue(), threadFactory); }{quote}
Below is the case about clean task after compaction. there nearly 3 hours
delay for removing file "mc-56525".
{quote}
TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664
LifecycleTransaction.java:363 - Staging for obsolescence
BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
..
TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 -
removing
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big from
list of files tracked for test_2.edgestore
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
before barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 -
Async instance tidier for
/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big,
completed
{quote}
was:
We use cassandra as backend storage for Janusgraph. when we loading huge data
(~2 billion vertex, ~10 billion edges), we met an problems.
At first, we use STCS as compaction strategy , but met below exception. we
checked the value of "max memory lock" is unlimited and "map count" is 1
million, these values should enough for loading. last , we found this problem
is caused by the virtual memory are all cosumed by cassandra. So not
additional virtual memory can be used by compaction task , and below exception
is thrower out.
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952
JVMStabilityInspector.javv
a:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the
virtual memory problem. But we found another problem : Many sstables which has
been compacted are still retained on disk, at last these old sstable consume so
many disk space, it's causing no enough disk for saving real data. we found
that so many files like
