[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203172#comment-15203172
]
DOAN DuyHai edited comment on CASSANDRA-11383 at 3/20/16 8:58 AM:
--
[~xedin]
Last update from the testing. I put the cluster in *ideal* conditions as you
recommended.
JVM settings:
- CMS
- Xmx8g, Xms8G
C* settings:
- concurrent_compactors: 6
Test conditions:
- cluster *idle* (no write, no read)
- LCS with *sstable_size_in_mb* = 1024 (1Gb)
- *no compaction ongoing* (took a whole night to compact for LCS)
- {{CREATE CUSTOM INDEX resource_period_end_month_int_idx ON
sharon.resource_bench (period_end_month_int) USING
'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = \{'mode':
'SPARSE'\};}}
Observations:
- I/O idle, CPU not exceeding 20% on average
(http://postimg.org/image/f664wm8dp/)
- {{nodetool compactionstats}} only show 1 index rebuild ongoing per node
{noformat}
id compaction type keyspace table
completed total unit progress
d8b4f4b0-ee6a-11e5-81f5-bd5584064785 Secondary index build sharon
resource_bench 9535985318 18920482745 bytes 50.40%
id compaction type keyspace table
completed total unit progress
d8b65440-ee6a-11e5-b44b-4deeb5ac98a3 Secondary index build sharon
resource_bench 9464081317 20988668046 bytes 45.09%
id compaction type keyspace table
completed total unit progress
d8b3bc30-ee6a-11e5-a152-db40f4fbe6b8 Secondary index build sharon
resource_bench 9471325678 17061191471 bytes 55.51%
id compaction type keyspace table
completed total unit progress
d8b45870-ee6a-11e5-b26b-53ed13e9667e Secondary index build sharon
resource_bench 9120598050 18921737677 bytes 48.20%
id compaction type keyspace table
completed total unit progress
d8b45870-ee6a-11e5-b2a3-331c04173c53 Secondary index build sharon
resource_bench 8943568835 20591008789 bytes 43.43%
id compaction type keyspace table
completed total unit progress
d8b47f80-ee6a-11e5-9fc8-0597212274c1 Secondary index build sharon
resource_bench 10172038156 21422242706 bytes 47.48%
id compaction type keyspace table
completed total unit progress
d8b34700-ee6a-11e5-a642-6dee841e75e5 Secondary index build sharon
resource_bench 10161205385 18730171060 bytes 54.25%
id compaction type keyspace table
completed total unit progress
d8b6f080-ee6a-11e5-8da4-bd70732fdab1 Secondary index build sharon
resource_bench 9961529350 21294352899 bytes 46.78%
id compaction type keyspace table
completed total unit progress
d8b43160-ee6a-11e5-8ac9-f59d626eedfa Secondary index build sharon
resource_bench 9160286080 22153527929 bytes 41.35%
id compaction type keyspace table
completed total unit progress
d8b51bc0-ee6a-11e5-8aa0-b9e611280aba Secondary index build sharon
resource_bench 9397690505 22791700212 bytes 41.23%
id compaction type keyspace table
completed total unit progress
d8b542d0-ee6a-11e5-8521-fbd14b018db6 Secondary index build sharon
resource_bench 10029096174 18910334578 bytes 53.04%
id compaction type keyspace table
completed total unit progress
d8b40a50-ee6a-11e5-a7b2-4b114ced0935 Secondary index build sharon
resource_bench 10118551269 16938426769 bytes 59.74%
id compaction type keyspace table
completed total unit progress
d8b039c0-ee6a-11e5-9a98-ff9a6f2af762 Secondary index build sharon
resource_bench 9003236945 18252472495 bytes 49.33%
{noformat}
- *there are still A LOT of GC* (with some ConcurrentMarkSweep lasting up to
10 secs!)
{noformat}
INFO [Service Thread] 2016-03-20 09:46:44,695 GCInspector.java:284 - ParNew GC
in 455ms. CMS Old Gen: 2964960608 -> 3487392640; Par Eden Space: 1006632960 ->
0;
INFO [Service Thread] 2016-03-20 09:46:47,250 GCInspector.java:284 - ParNew GC
in 460ms. CMS Old Gen: 3487392640 -> 3990379824; Par Eden Space: 1006632960 ->
0; Par Survivor Space: 125829120 -> 125828160
INFO [Service Thread] 2016-03-20 09:46:49,452 GCInspector.java:284 - ParNew GC
in 414ms. CMS Old Gen: 3990379824 -> 4445691424; Par Eden Space: 1006632960 ->
0; Par Survivor Space: 125828160 -> 125827840
INFO [Service Thread] 2016-03-20 09:46:52,328 GCInspector.java:284 - ParNew GC
in 484ms. CMS Old Gen: 4445691424
[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202088#comment-15202088
]
DOAN DuyHai edited comment on CASSANDRA-11383 at 3/18/16 8:30 PM:
--
[~jkrupan]
1. Not that large, see below the Spark script to generate randomized data:
{noformat}
import java.util.UUID
import com.datastax.spark.connector._
case class Resource(dsrId:UUID, relSeq:Long, seq:Long,
dspReleaseCode:String,
commercialOfferCode:String, transferCode:String,
mediaCode:String,
modelCode:String, unicWork:String,
title:String, status:String,
contributorsName:List[String],
periodEndMonthInt:Int, dspCode:String,
territoryCode:String,
payingNetQty:Long, authorizedSocietiesTxt: String,
relType:String)
val allDsps = List("youtube", "itunes", "spotify", "deezer", "vevo",
"google-play", "7digital", "spotify", "youtube", "spotify", "youtube",
"youtube", "youtube")
val allCountries = List("FR", "UK", "BE", "IT", "NL", "ES", "FR", "FR")
val allPeriodsEndMonths:Seq[Int] = for(year <- 2013 to 2015; month <- 1 to
12) yield (year.toString + f"$month%02d").toInt
val allModelCodes = List("PayAsYouGo", "AdFunded", "Subscription")
val allMediaCodes = List("Music","Ringtone")
val allTransferCodes = List("Streaming","Download")
val allCommercialOffers = List("Premium","Free")
val status = "Declared"
val authorizedSocietiesTxt: String="sacem sgae"
val relType = "whatever"
val titlesAndContributors: Array[(String, String)] =
sc.textFile("/tmp/top_100.csv").map(line => {val split = line.split(";");
(split(1),split(2))}).distinct.collect
for(i<- 1 to 100) {
sc.parallelize((1 to 4000).map(i => UUID.randomUUID)).
map(dsrId => {
val r = new java.util.Random(System.currentTimeMillis())
val relSeq = r.nextLong()
val seq = r.nextLong()
val dspReleaseCode = seq.toString
val dspCode = allDsps(r.nextInt(allDsps.size))
val periodEndMonth =
allPeriodsEndMonths(r.nextInt(allPeriodsEndMonths.size))
val territoryCode = allCountries(r.nextInt(allCountries.size))
val modelCode = allModelCodes(r.nextInt(allModelCodes.size))
val mediaCode = allMediaCodes(r.nextInt(allMediaCodes.size))
val transferCode =
allTransferCodes(r.nextInt(allTransferCodes.size))
val commercialOffer =
allCommercialOffers(r.nextInt(allCommercialOffers.size))
val titleAndContributor: (String, String) =
titlesAndContributors(r.nextInt(titlesAndContributors.size))
val title = titleAndContributor._1
val contributorsName = titleAndContributor._2.split(",").toList
val unicWork = title + "|" + titleAndContributor._2
val payingNetQty = r.nextInt(100).toLong
Resource(dsrId, relSeq, seq, dspReleaseCode, commercialOffer,
transferCode, mediaCode, modelCode,
unicWork, title, status, contributorsName, periodEndMonth,
dspCode, territoryCode, payingNetQty,
authorizedSocietiesTxt, relType)
}).
saveToCassandra("keyspace", "resource")
Thread.sleep(500)
}
{noformat}
2. Does OOM occur if SASI indexes are created one at a time - serially, waiting
for full index to build before moving on to the next? --> *Yes it does*, see
log file with CMS settings attached above
3. Do you need a 32G heap to build just one index? I cringe when I see a heap
larger than 14G. See if you can get a single SASI index build to work in 10-12G
or less.
--> Well the 32Gb heap was for analytics use-cases and I was using G1 GC. But
changing to CMS with 8Gb heap has the same result, OOM. see log file with CMS
settings attached above
was (Author: doanduyhai):
[~jkrupan]
1. Not that large, see below the Spark script to generate randomized data:
{code:scala}
import java.util.UUID
import com.datastax.spark.connector._
case class Resource(dsrId:UUID, relSeq:Long, seq:Long,
dspReleaseCode:String,
commercialOfferCode:String, transferCode:String,
mediaCode:String,
modelCode:String, unicWork:String,
title:String, status:String,
contributorsName:List[String],
periodEndMonthInt:Int, dspCode:String,
territoryCode:String,
payingNetQty:Long, authorizedSocietiesTxt: String,
relType:String)
val allDsps = List("youtube", "itunes", "spotify", "deezer", "vevo",
"google-play", "7digital", "spotify", "youtube", "spotify", "youtube",
"youtube", "youtube")
val allCountries = List("FR", "UK", "BE", "IT", "NL", "ES", "FR", "FR")
val allPeriodsEndMonths:Seq[Int] = for(year <- 2013 to 2015;
[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202851#comment-15202851
]
DOAN DuyHai edited comment on CASSANDRA-11383 at 3/19/16 5:15 PM:
--
[~jkrupan]
Other than terminology and wording/documentation about {{SPARSE}} mode, what
interests me more is how SASI can deal with {{DENSE}} index e.g. few indexed
value for millions/billions of matching primary keys.
The original secondary index was not adapted for
1. very low cardinality (index on email to search for user for example) because
it does not scale well with cluster size. In worst case you'll need to scan
N/RF nodes to fetch 0 or at most 1 user so the ratio effort vs result is bad
2. very high cardinality (user gender for example) because for each distinct
indexed value, you can have many matching users and it creates ultra wide-rows,
an anti-pattern
With SASI, although point 1. still holds (that's the common issue with all
**distributed** index systems, even Solr or ES) I had hoped that limitation 2.
will be lifted since SASI stores data in its own structures
One can argue that having an index on {{DENSE}} fields like country code (only
7-8 distinct values for the whole dataset) is a bad idea but they are meant to
be used in conjunction with other indices to cut down the matching dataset and
I rely on SASI query planner for this job
was (Author: doanduyhai):
[~jkrupan]
Other than terminology and wording/documentation about {{SPARSE}} mode, what
interests me more is how SASI can deal with {{DENSE}} index e.g. few indexed
value for millions/billions of matching primary keys.
The original secondary index was not adapted for
1. very low cardinality (index on email to search for user for example) because
it does not scale well with cluster size. In worst case you'll need to scan
N/RF nodes to fetch 0 or at most 1 user so the ratio effort vs result is bad
2. very high cardinality (user gender for example) because for each distinct
indexed value, you can have many matching users and it creates ultra wide-rows,
an anti-pattern
With SASI, although point 1. still holds (that's the common issue with all
**distributed** index systems, even Solr or ES) I had hoped that limitation 2.
will be lifted since SASI stores data in its own structures
> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log,
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
> JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
> - ≈ 100Gb/per node
> - 1.3 Tb cluster-wide
> - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
> - 8 indices with text field, NonTokenizingAnalyser, PREFIX mode,
> case-insensitive
> - 1 index with numeric field, SPARSE mode
> After a while, the nodes just gone OOM.
> I attach log files. You can see a lot of GC happening while index segments
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[ https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202816#comment-15202816 ] Jack Krupansky edited comment on CASSANDRA-11383 at 3/19/16 3:52 PM: - The terminology is a bit confusing here - everybody understands what a sparse matrix is, but exactly what constitutes sparseness in a column is very unclear. What is clear is that the cardinality (number of distinct values) is low for that int field. A naive person (okay... me) would have thought that sparse data meant few distinct values, which is what the int field is (36 distinct values.) I decided to check the doc to see what it says about SPARSE, but discovered that the doc doesn't exist yet in the main Cassandra doc - I sent a message to d...@datastax.com about that (turns out, they sync the doc to the DataStax Distribution of Cassandra (DDC) and DDC 3.4 is not out yet, coming soon.) So I went back to the orginal, pre-integration doc (https://github.com/xedin/sasi) and see that there is separate, non-integrated doc for SASI in the Cassandra source tree - https://github.com/apache/cassandra/blob/trunk/doc/SASI.md, which makes clear that "SPARSE, which is meant to improve performance of querying large, dense number ranges like timestamps for data inserted every millisecond." Oops... SPARSE=dense, but in any case SPARSE is designed for high cardinality of distinct values, which the int field is clearly not. I would argue that SASI should give a strongly-worded warning if the column data for a SPARSE index has low cardinality - low number of distinct column values and high number of index values per column value. was (Author: jkrupan): The terminology is a bit confusing here - everybody understands what a sparse matrix is, but exactly what constitutes sparseness in a column is very unclear. What is clear is that the cardinality (number of distinct values) is low for that int field. A naive person (okay... me) would have thought that sparse data meant few distinct values, which is what the int field is (36 distinct values.) I decided to check the doc to see what it says about SPARSE, but discovered that the doc doesn't exist yet in the main Cassandra doc - I sent a message to d...@datastax.com about that. So I went back to the orginal, pre-integration doc (https://github.com/xedin/sasi) and see that there is separate, non-integrated doc for SASI in the Cassandra source tree - https://github.com/apache/cassandra/blob/trunk/doc/SASI.md, which makes clear that "SPARSE, which is meant to improve performance of querying large, dense number ranges like timestamps for data inserted every millisecond." Oops... SPARSE=dense, but in any case SPARSE is designed for high cardinality of distinct values, which the int field is clearly not. I would argue that SASI should give a strongly-worded warning if the column data for a SPARSE index has low cardinality - low number of distinct column values and high number of index values per column value. > SASI index build leads to massive OOM > - > > Key: CASSANDRA-11383 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11383 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: C* 3.4 >Reporter: DOAN DuyHai > Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, > system.log_sasi_build_oom > > > 13 bare metal machines > - 6 cores CPU (12 HT) > - 64Gb RAM > - 4 SSD in RAID0 > JVM settings: > - G1 GC > - Xms32G, Xmx32G > Data set: > - ≈ 100Gb/per node > - 1.3 Tb cluster-wide > - ≈ 20Gb for all SASI indices > C* settings: > - concurrent_compactors: 1 > - compaction_throughput_mb_per_sec: 256 > - memtable_heap_space_in_mb: 2048 > - memtable_offheap_space_in_mb: 2048 > I created 9 SASI indices > - 8 indices with text field, NonTokenizingAnalyser, PREFIX mode, > case-insensitive > - 1 index with numeric field, SPARSE mode > After a while, the nodes just gone OOM. > I attach log files. You can see a lot of GC happening while index segments > are flush to disk. At some point the node OOM ... > /cc [~xedin] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202712#comment-15202712
]
DOAN DuyHai edited comment on CASSANDRA-11383 at 3/19/16 10:44 AM:
---
[~xedin]
bq. I've figured out what is going on and first of all period_end_month_int
index is not sparse - at least first term in that index has ~11M tokens
assigned to it
You're right, {{period_end_month_int}} is not *parse* in the sense we mean it
in English but SASI index mode {{SPARSE}} is the only one allowed for numeric
fields, {{PREFIX}} and {{CONTAINS}} are reserved to text fields. So we have a
fundamental issue here, how to index *dense* numeric values ?
bq. Temporary fix for this situation is switching to LCS with fixed maximum
sstable size, as I mentioned in my previous comment.
Can you elaborate further ? What, in LCS, makes it work with current situation
compared to STCS ? Is it the total number of SSTables ? (currently with STCS
there less than 100 SSTables per node so it's not really a big issue) Is it the
fact that a partition is guanrateed to be in a single SSTable with LCS ? (again
considering the schema we have mostly tiny rows but a lot of them)
For now I'm going to switch to LCS to see if we can finish building the index
without OOM. For long term, LCS is not the solution because this table size
will increase quickly over time and having tombstones in level > L3 will make
them rarely compacted
was (Author: doanduyhai):
bq. I've figured out what is going on and first of all period_end_month_int
index is not sparse - at least first term in that index has ~11M tokens
assigned to it
You're right, {{period_end_month_int}} is not *parse* in the sense we mean it
in English but SASI index mode {{SPARSE}} is the only one allowed for numeric
fields, {{PREFIX}} and {{CONTAINS}} are reserved to text fields. So we have a
fundamental issue here, how to index *dense* numeric values ?
bq. Temporary fix for this situation is switching to LCS with fixed maximum
sstable size, as I mentioned in my previous comment.
Can you elaborate further ? What, in LCS, makes it work with current situation
compared to STCS ? Is it the total number of SSTables ? (currently with STCS
there less than 100 SSTables per node so it's not really a big issue) Is it the
fact that a partition is guanrateed to be in a single SSTable with LCS ? (again
considering the schema we have mostly tiny rows but a lot of them)
For now I'm going to switch to LCS to see if we can finish building the index
without OOM. For long term, LCS is not the solution because this table size
will increase quickly over time and having tombstones in level > L3 will make
them rarely compacted
> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log,
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
> JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
> - ≈ 100Gb/per node
> - 1.3 Tb cluster-wide
> - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
> - 8 indices with text field, NonTokenizingAnalyser, PREFIX mode,
> case-insensitive
> - 1 index with numeric field, SPARSE mode
> After a while, the nodes just gone OOM.
> I attach log files. You can see a lot of GC happening while index segments
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[ https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202669#comment-15202669 ] Pavel Yaskevich edited comment on CASSANDRA-11383 at 3/19/16 9:09 AM: -- [~doanduyhai] I've figured out what is going on and first of all period_end_month_int index is not sparse - at least first term in that index has ~11M tokens assigned to it, that's where the source of the problem is - because it's sparse + composite combined TokenTreeBuilder has to pull a lot of stuff into memory when stitching segments together, I'm trying to figure out if there is a way to make it less memory intensive. Temporary fix for this situation is switching to LCS with fixed maximum sstable size, as I mentioned in my previous comment. was (Author: xedin): [~doanduyhai] I've figured out what is going on and first of all period_end_month_int index is not sparse - at least first term in that index has ~11M tokens assigned to it, that's where the source of the problem is - because it's sparse + composite combined TokenTreeBuilder has to pull a lot of stuff into memory when stitching segments together, I'm trying to figure out if there is a way to make it less memory intensive. > SASI index build leads to massive OOM > - > > Key: CASSANDRA-11383 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11383 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: C* 3.4 >Reporter: DOAN DuyHai > Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, > system.log_sasi_build_oom > > > 13 bare metal machines > - 6 cores CPU (12 HT) > - 64Gb RAM > - 4 SSD in RAID0 > JVM settings: > - G1 GC > - Xms32G, Xmx32G > Data set: > - ≈ 100Gb/per node > - 1.3 Tb cluster-wide > - ≈ 20Gb for all SASI indices > C* settings: > - concurrent_compactors: 1 > - compaction_throughput_mb_per_sec: 256 > - memtable_heap_space_in_mb: 2048 > - memtable_offheap_space_in_mb: 2048 > I created 9 SASI indices > - 8 indices with text field, NonTokenizingAnalyser, PREFIX mode, > case-insensitive > - 1 index with numeric field, SPARSE mode > After a while, the nodes just gone OOM. > I attach log files. You can see a lot of GC happening while index segments > are flush to disk. At some point the node OOM ... > /cc [~xedin] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202156#comment-15202156
]
DOAN DuyHai edited comment on CASSANDRA-11383 at 3/18/16 8:57 PM:
--
[~xedin]
I can't promis anything but I'm going to find a way to share data. We're
talking of 100Gb to upload ...
Below is the ma-1831 standard files:
{noformat}
root@ns3033877:~# ll
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-*
| grep -v "SI"
-rw-r--r-- 1 cassandra cassandra 4081363 Mar 17 21:40
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 16350922629 Mar 17 21:40
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-Data.db
-rw-r--r-- 1 cassandra cassandra 10 Mar 17 21:40
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 150496120 Mar 17 21:40
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 4678909890 Mar 17 21:40
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-Index.db
-rw-r--r-- 1 cassandra cassandra 12601 Mar 17 21:40
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra40410476 Mar 17 21:40
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 92 Mar 17 21:40
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-TOC.txt
{noformat}
And the SASI indices:
{noformat}
-rw-r--r-- 1 cassandra cassandra 97 Mar 18 20:57
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db
-rw-r--r-- 1 cassandra cassandra 24821784 Mar 18 20:20
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_0
-rw-r--r-- 1 cassandra cassandra 24825880 Mar 18 20:20
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_1
-rw-r--r-- 1 cassandra cassandra 24821784 Mar 18 20:25
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_10
-rw-r--r-- 1 cassandra cassandra 24817688 Mar 18 20:26
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_11
-rw-r--r-- 1 cassandra cassandra 24821784 Mar 18 20:26
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_12
-rw-r--r-- 1 cassandra cassandra 24821784 Mar 18 20:26
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_13
-rw-r--r-- 1 cassandra cassandra 24821784 Mar 18 20:27
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_14
-rw-r--r-- 1 cassandra cassandra 24817688 Mar 18 20:27
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_15
-rw-r--r-- 1 cassandra cassandra 24825880 Mar 18 20:27
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_16
-rw-r--r-- 1 cassandra cassandra 24825880 Mar 18 20:28
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_17
-rw-r--r-- 1 cassandra cassandra 24825880 Mar 18 20:28
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_18
-rw-r--r-- 1 cassandra cassandra 24817688 Mar 18 20:29
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_19
-rw-r--r-- 1 cassandra cassandra 24821784 Mar 18 20:21
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_2
-rw-r--r-- 1 cassandra cassandra 24821784 Mar 18 20:29
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_20
-rw-r--r-- 1 cassandra cassandra 24825880 Mar 18 20:30
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_21
-rw-r--r-- 1 cassandra cassandra 24821784 Mar 18 20:30
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_dsp_code_idx.db_22
-rw-r--r-- 1 cassandra cassandra 24817688 Mar 18 20:31
[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202088#comment-15202088
]
DOAN DuyHai edited comment on CASSANDRA-11383 at 3/18/16 8:31 PM:
--
[~jkrupan]
1. Not that large, see below the Spark script to generate randomized data:
{noformat}
import java.util.UUID
import com.datastax.spark.connector._
case class Resource(dsrId:UUID, relSeq:Long, seq:Long,
dspReleaseCode:String,
commercialOfferCode:String, transferCode:String,
mediaCode:String,
modelCode:String, unicWork:String,
title:String, status:String,
contributorsName:List[String],
periodEndMonthInt:Int, dspCode:String,
territoryCode:String,
payingNetQty:Long, authorizedSocietiesTxt: String,
relType:String)
val allDsps = List("youtube", "itunes", "spotify", "deezer", "vevo",
"google-play", "7digital", "spotify", "youtube", "spotify", "youtube",
"youtube", "youtube")
val allCountries = List("FR", "UK", "BE", "IT", "NL", "ES", "FR", "FR")
val allPeriodsEndMonths:Seq[Int] = for(year <- 2013 to 2015; month <- 1 to
12) yield (year.toString + f"$month%02d").toInt
val allModelCodes = List("PayAsYouGo", "AdFunded", "Subscription")
val allMediaCodes = List("Music","Ringtone")
val allTransferCodes = List("Streaming","Download")
val allCommercialOffers = List("Premium","Free")
val status = "Declared"
val authorizedSocietiesTxt: String="sacem sgae"
val relType = "whatever"
val titlesAndContributors: Array[(String, String)] =
sc.textFile("/tmp/top_100.csv").map(line => {val split = line.split(";");
(split(1),split(2))}).distinct.collect
for(i<- 1 to 100) {
sc.parallelize((1 to 4000).map(i => UUID.randomUUID)).
map(dsrId => {
val r = new java.util.Random(System.currentTimeMillis())
val relSeq = r.nextLong()
val seq = r.nextLong()
val dspReleaseCode = seq.toString
val dspCode = allDsps(r.nextInt(allDsps.size))
val periodEndMonth =
allPeriodsEndMonths(r.nextInt(allPeriodsEndMonths.size))
val territoryCode = allCountries(r.nextInt(allCountries.size))
val modelCode = allModelCodes(r.nextInt(allModelCodes.size))
val mediaCode = allMediaCodes(r.nextInt(allMediaCodes.size))
val transferCode =
allTransferCodes(r.nextInt(allTransferCodes.size))
val commercialOffer =
allCommercialOffers(r.nextInt(allCommercialOffers.size))
val titleAndContributor: (String, String) =
titlesAndContributors(r.nextInt(titlesAndContributors.size))
val title = titleAndContributor._1
val contributorsName = titleAndContributor._2.split(",").toList
val unicWork = title + "|" + titleAndContributor._2
val payingNetQty = r.nextInt(100).toLong
Resource(dsrId, relSeq, seq, dspReleaseCode, commercialOffer,
transferCode, mediaCode, modelCode,
unicWork, title, status, contributorsName, periodEndMonth,
dspCode, territoryCode, payingNetQty,
authorizedSocietiesTxt, relType)
}).
saveToCassandra("keyspace", "resource")
Thread.sleep(500)
}
{noformat}
The indices
{noformat}
CREATE CUSTOM INDEX resource_territory_code_idx ON sharon.resource_bench
(territory_code) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS
= {'analyzer_class':
'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer',
'case_sensitive': 'false'};
CREATE CUSTOM INDEX resource_dsp_code_idx ON sharon.resource_bench (dsp_code)
USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS =
{'analyzer_class':
'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer',
'case_sensitive': 'false'};
CREATE CUSTOM INDEX resource_commercial_offer_code_idx ON sharon.resource_bench
(commercial_offer_code) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH
OPTIONS = {'analyzer_class':
'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer',
'case_sensitive': 'false'};
CREATE CUSTOM INDEX resource_authorized_societies_txt_idx ON
sharon.resource_bench (authorized_societies_txt) USING
'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {'analyzer_class':
'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
'tokenization_normalize_lowercase': 'true', 'mode': 'PREFIX', 'analyzed':
'true', 'tokenization_enable_stemming': 'true'};
CREATE CUSTOM INDEX resource_transfer_code_idx ON sharon.resource_bench
(transfer_code) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS
= {'analyzer_class':
'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer',
'case_sensitive': 'false'};
CREATE CUSTOM INDEX resource_rel_type_idx ON
[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202259#comment-15202259
]
DOAN DuyHai edited comment on CASSANDRA-11383 at 3/18/16 10:16 PM:
---
[~xedin]
OK, I'm trying to fetch the sstable with its data files
In the meantime, I just re-create one index as shown above using
*max_compaction_flush_memory_in_mb* = 128 and SASI flushes thousands of index
files (of ~16kb each) and eventually the server dies maybe because too many
file handles
{noformat}
INFO [SASI-General:1] 2016-03-18 22:45:37,480 PerSSTableIndexWriter.java:258 -
Flushed index segment
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130370,
took 0 ms.
INFO [SASI-General:1] 2016-03-18 22:45:37,581 PerSSTableIndexWriter.java:258 -
Flushed index segment
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130371,
took 101 ms.
ERROR [SASI-General:1] 2016-03-18 22:45:37,582 CassandraDaemon.java:195 -
Exception in thread Thread[SASI-General:1,5,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.index.sasi.utils.MappedBuffer.(MappedBuffer.java:78)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.index.sasi.utils.MappedBuffer.(MappedBuffer.java:57)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndex.(OnDiskIndex.java:142)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$scheduleSegmentFlush$260(PerSSTableIndexWriter.java:253)
~[apache-cassandra-3.4.jar:3.4]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_74]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_74]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
~[na:1.8.0_74]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
~[apache-cassandra-3.4.jar:3.4]
... 8 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
{noformat}
was (Author: doanduyhai):
[~xedin]
OK, I'm trying to fetch the sstable with its data files
In the meantime, I just re-create one index as shown above using
*max_compaction_flush_memory_in_mb* = 128 and SASI flushes thousands of index
files (of ~16kb each) and eventually the server dies maybe because too many
file handles
{noformat}
INFO [SASI-General:1] 2016-03-18 22:45:37,480 PerSSTableIndexWriter.java:258 -
Flushed index segment
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130370,
took 0 ms.
INFO [SASI-General:1] 2016-03-18 22:45:37,581 PerSSTableIndexWriter.java:258 -
Flushed index segment
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130371,
took 101 ms.
ERROR [SASI-General:1] 2016-03-18 22:45:37,582 CassandraDaemon.java:195 -
Exception in thread Thread[SASI-General:1,5,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
{noformat}
> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log,
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
> JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
> - ≈ 100Gb/per node
> - 1.3 Tb cluster-wide
> - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
> - 8 indices with text field, NonTokenizingAnalyser, PREFIX mode,
> case-insensitive
> - 1 index with numeric field, SPARSE mode
> After a while, the nodes just gone OOM.
> I attach log files. You can see a lot of GC happening while index segments
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202259#comment-15202259
]
DOAN DuyHai edited comment on CASSANDRA-11383 at 3/18/16 10:11 PM:
---
[~xedin]
OK, I'm trying to fetch the sstable with its data files
In the meantime, I just re-create one index as shown above using
*max_compaction_flush_memory_in_mb* = 128 and SASI flushes thousands of index
files (of ~16kb each) and eventually the server dies maybe because too many
file handles
{noformat}
INFO [SASI-General:1] 2016-03-18 22:45:37,480 PerSSTableIndexWriter.java:258 -
Flushed index segment
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130370,
took 0 ms.
INFO [SASI-General:1] 2016-03-18 22:45:37,581 PerSSTableIndexWriter.java:258 -
Flushed index segment
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130371,
took 101 ms.
ERROR [SASI-General:1] 2016-03-18 22:45:37,582 CassandraDaemon.java:195 -
Exception in thread Thread[SASI-General:1,5,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
{noformat}
was (Author: doanduyhai):
[~xedin]
OK, I'm trying to fetch the sstable with its data files
In the meantime, I just re-create one index as shown above using *
max_compaction_flush_memory_in_mb * = 128 and SASI flushes thousands of index
files and eventually the server dies maybe because too many file handles
{noformat}
INFO [SASI-General:1] 2016-03-18 22:45:37,480 PerSSTableIndexWriter.java:258 -
Flushed index segment
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130370,
took 0 ms.
INFO [SASI-General:1] 2016-03-18 22:45:37,581 PerSSTableIndexWriter.java:258 -
Flushed index segment
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130371,
took 101 ms.
ERROR [SASI-General:1] 2016-03-18 22:45:37,582 CassandraDaemon.java:195 -
Exception in thread Thread[SASI-General:1,5,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
{noformat}
> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log,
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
> JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
> - ≈ 100Gb/per node
> - 1.3 Tb cluster-wide
> - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
> - 8 indices with text field, NonTokenizingAnalyser, PREFIX mode,
> case-insensitive
> - 1 index with numeric field, SPARSE mode
> After a while, the nodes just gone OOM.
> I attach log files. You can see a lot of GC happening while index segments
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202262#comment-15202262
]
DOAN DuyHai edited comment on CASSANDRA-11383 at 3/18/16 10:13 PM:
---
[~jkrupan]
Below is the schema:
{noformat}
create table if not exists sharon.resource_bench (
dsr_id uuid,
rel_seq bigint,
seq bigint,
dsp_code varchar,
model_code varchar,
media_code varchar,
transfer_code varchar,
commercial_offer_code varchar,
territory_code varchar,
period_end_month_int int,
authorized_societies_txt text,
rel_type text,
status text,
dsp_release_code text,
title text,
contributors_name list,
unic_work text,
paying_net_qty bigint,
PRIMARY KEY ((dsr_id, rel_seq), seq)
) WITH CLUSTERING ORDER BY (seq ASC);
{noformat}
was (Author: doanduyhai):
[~jkrupan]
Below is the schema:
{code:sql}
create table if not exists sharon.resource_bench (
dsr_id uuid,
rel_seq bigint,
seq bigint,
dsp_code varchar,
model_code varchar,
media_code varchar,
transfer_code varchar,
commercial_offer_code varchar,
territory_code varchar,
period_end_month_int int,
authorized_societies_txt text,
rel_type text,
status text,
dsp_release_code text,
title text,
contributors_name list,
unic_work text,
paying_net_qty bigint,
PRIMARY KEY ((dsr_id, rel_seq), seq)
) WITH CLUSTERING ORDER BY (seq ASC);
{code:sql}
> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log,
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
> JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
> - ≈ 100Gb/per node
> - 1.3 Tb cluster-wide
> - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
> - 8 indices with text field, NonTokenizingAnalyser, PREFIX mode,
> case-insensitive
> - 1 index with numeric field, SPARSE mode
> After a while, the nodes just gone OOM.
> I attach log files. You can see a lot of GC happening while index segments
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
