[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-12-13 Thread Sam Tunnicliffe (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-12796:

Reviewer: Jeremiah Jordan  (was: Sam Tunnicliffe)

> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Assignee: Sam Tunnicliffe
>Priority: Critical
> Fix For: 2.2.9, 3.0.11, 3.10
>
>
> We have a table with rather wide partition and a secondary index defined over 
> it. As soon as we try to rebuild the index we observed exhaustion of Java 
> heap and eventual OOM error. After a lengthy investigation we have managed to 
> find a culprit which appears to be a wrong granule of barrier issuances in 
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table 
> which poses a problem for wide partition tables as flush runnable 
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed 
> with flushing secondary index memtable before completing operations prior 
> recent issue of the barrier. In our situation the flush runnable waits until 
> whole wide partition gets indexed into the secondary index memtable before 
> flushing it. This causes an exhaustion of the heap and eventual OOM error.
> After we changed granule of barrier issue in method 
> {{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to 
> table partition secondary index (see 
> [https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
>  rebuild started to work without heap exhaustion. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-12-13 Thread Sam Tunnicliffe (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-12796:

   Resolution: Fixed
Fix Version/s: 3.10
   3.0.11
   2.2.9
Reproduced In: 3.0.8, 2.2.7, 2.1.9  (was: 2.1.9, 2.2.7, 3.0.8)
   Status: Resolved  (was: Patch Available)

Thanks, committed to 2.2 in {{fb2940050e27a5642a23f3e9b5aaa7ae65e018b1}} and to 
3.0 (with the additional log info) in 
{{36ce4e02b429b1297d71c5c8a963623c62d9e159}}. Also added the new -D switch to 
{{jvm.options}} from 3.10 (i.e. the cassandra-3.11 branch) onwards.

> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
> Fix For: 2.2.9, 3.0.11, 3.10
>
>
> We have a table with rather wide partition and a secondary index defined over 
> it. As soon as we try to rebuild the index we observed exhaustion of Java 
> heap and eventual OOM error. After a lengthy investigation we have managed to 
> find a culprit which appears to be a wrong granule of barrier issuances in 
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table 
> which poses a problem for wide partition tables as flush runnable 
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed 
> with flushing secondary index memtable before completing operations prior 
> recent issue of the barrier. In our situation the flush runnable waits until 
> whole wide partition gets indexed into the secondary index memtable before 
> flushing it. This causes an exhaustion of the heap and eventual OOM error.
> After we changed granule of barrier issue in method 
> {{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to 
> table partition secondary index (see 
> [https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
>  rebuild started to work without heap exhaustion. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-11-10 Thread Milan Majercik (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Milan Majercik updated CASSANDRA-12796:
---
Reproduced In: 3.0.8, 2.2.7, 2.1.9  (was: 2.1.9, 2.2.7, 3.0.8)
Since Version: 2.2.7
   Status: Patch Available  (was: Awaiting Feedback)

> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
>
> We have a table with rather wide partition and a secondary index defined over 
> it. As soon as we try to rebuild the index we observed exhaustion of Java 
> heap and eventual OOM error. After a lengthy investigation we have managed to 
> find a culprit which appears to be a wrong granule of barrier issuances in 
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table 
> which poses a problem for wide partition tables as flush runnable 
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed 
> with flushing secondary index memtable before completing operations prior 
> recent issue of the barrier. In our situation the flush runnable waits until 
> whole wide partition gets indexed into the secondary index memtable before 
> flushing it. This causes an exhaustion of the heap and eventual OOM error.
> After we changed granule of barrier issue in method 
> {{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to 
> table partition secondary index (see 
> [https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
>  rebuild started to work without heap exhaustion. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-11-04 Thread Joshua McKenzie (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-12796:

Reproduced In: 3.0.8, 2.2.7, 2.1.9  (was: 2.1.9, 2.2.7, 3.0.8)
 Reviewer: Sam Tunnicliffe

> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
>
> We have a table with rather wide partition and a secondary index defined over 
> it. As soon as we try to rebuild the index we observed exhaustion of Java 
> heap and eventual OOM error. After a lengthy investigation we have managed to 
> find a culprit which appears to be a wrong granule of barrier issuances in 
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table 
> which poses a problem for wide partition tables as flush runnable 
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed 
> with flushing secondary index memtable before completing operations prior 
> recent issue of the barrier. In our situation the flush runnable waits until 
> whole wide partition gets indexed into the secondary index memtable before 
> flushing it. This causes an exhaustion of the heap and eventual OOM error.
> After we changed granule of barrier issue in method 
> {{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to 
> table partition secondary index (see 
> [https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
>  rebuild started to work without heap exhaustion. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-11-04 Thread anmols (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anmols updated CASSANDRA-12796:
---
Reproduced In: 3.0.8, 2.2.7, 2.1.9  (was: 2.2.7)
   Status: Awaiting Feedback  (was: Open)

> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
>
> We have a table with rather wide partition and a secondary index defined over 
> it. As soon as we try to rebuild the index we observed exhaustion of Java 
> heap and eventual OOM error. After a lengthy investigation we have managed to 
> find a culprit which appears to be a wrong granule of barrier issuances in 
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table 
> which poses a problem for wide partition tables as flush runnable 
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed 
> with flushing secondary index memtable before completing operations prior 
> recent issue of the barrier. In our situation the flush runnable waits until 
> whole wide partition gets indexed into the secondary index memtable before 
> flushing it. This causes an exhaustion of the heap and eventual OOM error.
> After we changed granule of barrier issue in method 
> {{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to 
> table partition secondary index (see 
> [https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
>  rebuild started to work without heap exhaustion. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-10-17 Thread Milan Majercik (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Milan Majercik updated CASSANDRA-12796:
---
Reproduced In: 2.2.7  (was: 2.2.x)

> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
>
> We have a table with rather wide partition and a secondary index defined over 
> it. As soon as we try to rebuild the index we observed exhaustion of Java 
> heap and eventual OOM error. After a lengthy investigation we have managed to 
> find a culprit which appears to be a wrong granule of barrier issuances in 
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table 
> which poses a problem for wide partition tables as flush runnable 
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed 
> with flushing secondary index memtable before completing operations prior 
> recent issue of the barrier. In our situation the flush runnable waits until 
> whole wide partition gets indexed into the secondary index memtable before 
> flushing it. This causes an exhaustion of the heap and eventual OOM error.
> After we changed granule of barrier issue in method 
> {{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to 
> table partition secondary index (see 
> [https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
>  rebuild started to work without heap exhaustion. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-10-17 Thread Milan Majercik (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Milan Majercik updated CASSANDRA-12796:
---
Since Version:   (was: 2.2.7)

> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
>
> We have a table with rather wide partition and a secondary index defined over 
> it. As soon as we try to rebuild the index we observed exhaustion of Java 
> heap and eventual OOM error. After a lengthy investigation we have managed to 
> find a culprit which appears to be a wrong granule of barrier issuances in 
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table 
> which poses a problem for wide partition tables as flush runnable 
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed 
> with flushing secondary index memtable before completing operations prior 
> recent issue of the barrier. In our situation the flush runnable waits until 
> whole wide partition gets indexed into the secondary index memtable before 
> flushing it. This causes an exhaustion of the heap and eventual OOM error.
> After we changed granule of barrier issue in method 
> {{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to 
> table partition secondary index (see 
> [https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
>  rebuild started to work without heap exhaustion. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-10-17 Thread Milan Majercik (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Milan Majercik updated CASSANDRA-12796:
---
Description: 
We have a table with rather wide partition and a secondary index defined over 
it. As soon as we try to rebuild the index we observed exhaustion of Java heap 
and eventual OOM error. After a lengthy investigation we have managed to find a 
culprit which appears to be a wrong granule of barrier issuances in method 
{{org.apache.cassandra.db.Keyspace.indexRow}}:

{code}
try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
{
Set indexes = 
cfs.indexManager.getIndexesByNames(idxNames);

Iterator pager = QueryPagers.pageRowLocally(cfs, 
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
}
{code}

Please note the operation group granule is a partition of the source table 
which poses a problem for wide partition tables as flush runnable 
({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed with 
flushing secondary index memtable before completing operations prior recent 
issue of the barrier. In our situation the flush runnable waits until whole 
wide partition gets indexed into the secondary index memtable before flushing 
it. This causes an exhaustion of the heap and eventual OOM error.

After we changed granule of barrier issue in method 
{{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to table 
partition secondary index (see 
[https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
 rebuild started to work without heap exhaustion. 

  was:
We have a table with rather wide partition with a secondary index defined over 
it. As soon as we try to rebuild the index we observed exhaustion of Java heap 
and eventual OOM error. After a lengthy investigation we have managed to find a 
culprit which appears to be a wrong granule of barrier issuances in method 
{{org.apache.cassandra.db.Keyspace.indexRow}}:

{code}
try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
{
Set indexes = 
cfs.indexManager.getIndexesByNames(idxNames);

Iterator pager = QueryPagers.pageRowLocally(cfs, 
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
}
{code}

Please note the operation group granule is a partition of the source table 
which poses a problem for wide partition tables as flush runnable 
({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed with 
flushing secondary index memtable before completing operations prior recent 
issue of the barrier. In our situation the flush runnable waits until whole 
wide partition gets indexed into the secondary index memtable before flushing 
it. This causes an exhaustion of the heap and eventual OOM error.

After we changed granule of barrier issue in method 
{{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to table 
partition secondary index (see 
[https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
 rebuild started to work without heap exhaustion. 


> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
>
> We have a table with rather wide partition and a secondary index defined over 
> it. As soon as we try to rebuild the index we observed exhaustion of Java 
> heap and eventual OOM error. After a lengthy investigation we have managed to 
> find a culprit which appears to be a wrong granule of barrier issuances in 
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {

[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-10-17 Thread Milan Majercik (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Milan Majercik updated CASSANDRA-12796:
---
Description: 
We have a table with rather wide partition with a secondary index defined over 
it. As soon as we try to rebuild the index we observed exhaustion of Java heap 
and eventual OOM error. After a lengthy investigation we have managed to find a 
culprit which appears to be a wrong granule of barrier issuances in method 
{{org.apache.cassandra.db.Keyspace.indexRow}}:

{code}
try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
{
Set indexes = 
cfs.indexManager.getIndexesByNames(idxNames);

Iterator pager = QueryPagers.pageRowLocally(cfs, 
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
}
{code}

Please note the operation group granule is a partition of the source table 
which poses a problem for wide partition tables as flush runnable 
({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed with 
flushing secondary index memtable before completing operations prior recent 
issue of the barrier. In our situation the flush runnable waits until whole 
wide partition gets indexed into the secondary index memtable before flushing 
it. This causes an exhaustion of the heap and eventual OOM error.

After we changed granule of barrier issue in method 
{{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to table 
partition secondary index (see 
[https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
 rebuild started to work without heap exhaustion. 

  was:
We have a table with rather wide partition with a secondary index defined over 
it. As soon as we try to rebuild the index we observed exhaustion of Java heap 
and eventual OOM error. After a lengthy investigation we have managed to find a 
culprit which appears to be a wrong granule of barrier issuances in method 
{{org.apache.cassandra.db.Keyspace.indexRow}}:

{code}
try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
{
Set indexes = 
cfs.indexManager.getIndexesByNames(idxNames);

Iterator pager = QueryPagers.pageRowLocally(cfs, 
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
}
{code}

Please note the operation group granule is a partition of the source table 
which poses a problem for wide partition tables as flush runnable 
({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed with 
flushing secondary index memtable before completing operations prior recent 
issue of the barrier. In our situation the flush runnable waits until whole 
wide partition gets indexed into the secondary index memtable before flushing 
it causing an exhaustion of the heap and eventual OOM error.


> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
>
> We have a table with rather wide partition with a secondary index defined 
> over it. As soon as we try to rebuild the index we observed exhaustion of 
> Java heap and eventual OOM error. After a lengthy investigation we have 
> managed to find a culprit which appears to be a wrong granule of barrier 
> issuances in method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = 

[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-10-17 Thread Milan Majercik (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Milan Majercik updated CASSANDRA-12796:
---
Description: 
We have a table with rather wide partition with a secondary index defined over 
it. As soon as we try to rebuild the index we observed exhaustion of Java heap 
and eventual OOM error. After a lengthy investigation we have managed to find a 
culprit which appears to be a wrong granule of barrier issuances in method 
{{org.apache.cassandra.db.Keyspace.indexRow}}:

{code}
try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
{
Set indexes = 
cfs.indexManager.getIndexesByNames(idxNames);

Iterator pager = QueryPagers.pageRowLocally(cfs, 
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
}
{code}

Please note the operation group granule is a partition of the source table 
which poses a problem for wide partition tables as flush runnable 
({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed with 
flushing secondary index memtable before completing operations prior recent 
issue of the barrier. In our situation the flush runnable waits until whole 
wide partition gets indexed into the secondary index memtable before flushing 
it causing an exhaustion of the heap and eventual OOM error.

  was:
We have a table with rather wide partition with a secondary index defined over 
it. As soon as we try to rebuild the index we observed exhaustion of Java heap 
and eventual OOM error. After a lengthy investigation we have managed to find a 
culprit which appears to be a wrong granule of barrier issuances in method 
{{org.apache.cassandra.db.Keyspace.indexRow}}:
{code}
{html}try (OpOrder.Group opGroup = 
cfs.keyspace.writeOrder.start()){html}
{
Set indexes = 
cfs.indexManager.getIndexesByNames(idxNames);

Iterator pager = QueryPagers.pageRowLocally(cfs, 
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
}
{code}



> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
>
> We have a table with rather wide partition with a secondary index defined 
> over it. As soon as we try to rebuild the index we observed exhaustion of 
> Java heap and eventual OOM error. After a lengthy investigation we have 
> managed to find a culprit which appears to be a wrong granule of barrier 
> issuances in method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table 
> which poses a problem for wide partition tables as flush runnable 
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed 
> with flushing secondary index memtable before completing operations prior 
> recent issue of the barrier. In our situation the flush runnable waits until 
> whole wide partition gets indexed into the secondary index memtable before 
> flushing it causing an exhaustion of the heap and 

[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-10-17 Thread Milan Majercik (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Milan Majercik updated CASSANDRA-12796:
---
Description: 
We have a table with rather wide partition with a secondary index defined over 
it. As soon as we try to rebuild the index we observed exhaustion of Java heap 
and eventual OOM error. After a lengthy investigation we have managed to find a 
culprit which appears to be a wrong granule of barrier issuances in method 
{{org.apache.cassandra.db.Keyspace.indexRow}}:
{code}
{html}try (OpOrder.Group opGroup = 
cfs.keyspace.writeOrder.start()){html}
{
Set indexes = 
cfs.indexManager.getIndexesByNames(idxNames);

Iterator pager = QueryPagers.pageRowLocally(cfs, 
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
}
{code}


  was:
We have a table with rather wide partition with a secondary index defined over 
it. As soon as we try to rebuild the index we observed exhaustion of Java heap 
and eventual OOM error. After a lengthy investigation we have managed to find a 
culprit which appears to be a wrong granule of barrier issuances in method 
{{org.apache.cassandra.db.Keyspace.indexRow}}:
{code}
*try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start())*
{
Set indexes = 
cfs.indexManager.getIndexesByNames(idxNames);

Iterator pager = QueryPagers.pageRowLocally(cfs, 
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
*}*
{code}



> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
>
> We have a table with rather wide partition with a secondary index defined 
> over it. As soon as we try to rebuild the index we observed exhaustion of 
> Java heap and eventual OOM error. After a lengthy investigation we have 
> managed to find a culprit which appears to be a wrong granule of barrier 
> issuances in method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> {html}try (OpOrder.Group opGroup = 
> cfs.keyspace.writeOrder.start()){html}
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12796) Heap exhaustion when rebuilding secondary index over a table with wide partitions

2016-10-17 Thread Milan Majercik (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Milan Majercik updated CASSANDRA-12796:
---
Description: 
We have a table with rather wide partition with a secondary index defined over 
it. As soon as we try to rebuild the index we observed exhaustion of Java heap 
and eventual OOM error. After a lengthy investigation we have managed to find a 
culprit which appears to be a wrong granule of barrier issuances in method 
{{org.apache.cassandra.db.Keyspace.indexRow}}:
{code}
*try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start())*
{
Set indexes = 
cfs.indexManager.getIndexesByNames(idxNames);

Iterator pager = QueryPagers.pageRowLocally(cfs, 
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
*}*
{code}


> Heap exhaustion when rebuilding secondary index over a table with wide 
> partitions
> -
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Milan Majercik
>Priority: Critical
>
> We have a table with rather wide partition with a secondary index defined 
> over it. As soon as we try to rebuild the index we observed exhaustion of 
> Java heap and eventual OOM error. After a lengthy investigation we have 
> managed to find a culprit which appears to be a wrong granule of barrier 
> issuances in method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> *try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start())*
> {
> Set indexes = 
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator pager = QueryPagers.pageRowLocally(cfs, 
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> *}*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)