date:20110310


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005015#comment-13005015
 ] 

Sylvain Lebresne commented on CASSANDRA-2104:
-

Can you open a new ticket with the description of the problem then. If you're 
at liberty to attach an bad sstable to reproduce the problem, that would 
clearly help.

 IndexOutOfBoundsException during lazy row compaction of supercolumns
 

 Key: CASSANDRA-2104
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2104
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.7.0
Reporter: Daniel Lundin
Assignee: Sylvain Lebresne
 Fix For: 0.7.3

 Attachments: 
 0001-Use-the-right-comparator-when-deserializing-superCol.patch, 
 Unit-test.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 I ran into an exception when lazily compacting wide rows of TimeUUID columns.
 It seems to trigger when a row is larger than 
 {{in_memory_compaction_limit_in_mb}}.
 Traceback:
 {noformat}
  INFO [CompactionExecutor:1] 2011-02-03 10:59:59,262 CompactionIterator.java 
 (line 135) Compacting large row X (76999384 bytes) incrementally
  ERROR [CompactionExecutor:1] 2011-02-03 10:59:59,266 
 AbstractCassandraDaemon.java (line 114) Fatal exception in thread T
  hread[CompactionExecutor:1,1,main]
  java.lang.IndexOutOfBoundsException
  at java.nio.Buffer.checkIndex(Buffer.java:514)
  at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
  at 
 org.apache.cassandra.db.marshal.TimeUUIDType.compareTimestampBytes(TimeUUIDType.java:56)
  at 
 org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:45)
  at 
 org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:29)
  at 
 java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(ConcurrentSkipListMap.java:606
  )
  at 
 java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:685)
  at 
 java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:864)
  at 
 java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1893)
  at 
 org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:170)
  at 
 org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:195)
  at 
 org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:221)
  at 
 org.apache.cassandra.io.LazilyCompactedRow$LazyColumnIterator.reduce(LazilyCompactedRow.java:204)
  at 
 org.apache.cassandra.io.LazilyCompactedRow$LazyColumnIterator.reduce(LazilyCompactedRow.java:185)
  at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:62)
  at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
  at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
  at 
 com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
  at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
  at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
  at 
 org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
  at 
 org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
  at 
 org.apache.cassandra.io.LazilyCompactedRow.init(LazilyCompactedRow.java:88)
  at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:137)
  at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
  at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
  at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
  at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
  at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
  at 
 org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
  at 
 org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
  at 
 org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:426)
  at 
 org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
  at 
 org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
  at

[jira] Updated: (CASSANDRA-1938) Use UUID as node identifiers in counters instead of IP addresses


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-1938:


Attachment: (was: 0002-Merge-old-shard-locally.patch)

 Use UUID as node identifiers in counters instead of IP addresses 
 -

 Key: CASSANDRA-1938
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1938
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
 Fix For: 0.8

 Attachments: 1938-rebase-and-fix-v1.txt, 1938_discussion

   Original Estimate: 56h
  Remaining Estimate: 56h

 The use of IP addresses as node identifiers in the partition of a given
 counter is fragile. Changes of the node's IP addresses can result in data
 loss. This patch proposes to use UUIDs instead.
 NOTE: this breaks the on-disk file format (for counters)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-1938) Use UUID as node identifiers in counters instead of IP addresses


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-1938:


Attachment: (was: 0001-Use-uuid-instead-of-IP-for-counters.patch)

 Use UUID as node identifiers in counters instead of IP addresses 
 -

 Key: CASSANDRA-1938
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1938
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
 Fix For: 0.8

 Attachments: 1938-rebase-and-fix-v1.txt, 1938_discussion

   Original Estimate: 56h
  Remaining Estimate: 56h

 The use of IP addresses as node identifiers in the partition of a given
 counter is fragile. Changes of the node's IP addresses can result in data
 loss. This patch proposes to use UUIDs instead.
 NOTE: this breaks the on-disk file format (for counters)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-1938) Use UUID as node identifiers in counters instead of IP addresses


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-1938:


Attachment: (was: 0003-Thrift-change-to-CfDef.patch)

 Use UUID as node identifiers in counters instead of IP addresses 
 -

 Key: CASSANDRA-1938
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1938
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
 Fix For: 0.8

 Attachments: 1938-rebase-and-fix-v1.txt, 1938_discussion

   Original Estimate: 56h
  Remaining Estimate: 56h

 The use of IP addresses as node identifiers in the partition of a given
 counter is fragile. Changes of the node's IP addresses can result in data
 loss. This patch proposes to use UUIDs instead.
 NOTE: this breaks the on-disk file format (for counters)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (CASSANDRA-2306) generate endpointList depending on command type (read or write)

generate endpointList depending on command type (read or write)
---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-1938) Use UUID as node identifiers in counters instead of IP addresses


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-1938:


Attachment: 0003-Thrift-change-to-CfDef.patch
0002-Merge-old-shard-locally.patch
0001-Use-uuid-instead-of-IP-for-counters.patch

Rebased to current trunk (svn rev 1080098)

I'll also try to keep 
https://github.com/pcmanus/cassandra/tree/1938_counters_uuid updated (aka, 
rebased to trunk).

 Use UUID as node identifiers in counters instead of IP addresses 
 -

 Key: CASSANDRA-1938
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1938
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
 Fix For: 0.8

 Attachments: 0001-Use-uuid-instead-of-IP-for-counters.patch, 
 0002-Merge-old-shard-locally.patch, 0003-Thrift-change-to-CfDef.patch, 
 1938-rebase-and-fix-v1.txt, 1938_discussion

   Original Estimate: 56h
  Remaining Estimate: 56h

 The use of IP addresses as node identifiers in the partition of a given
 counter is fragile. Changes of the node's IP addresses can result in data
 loss. This patch proposes to use UUIDs instead.
 NOTE: this breaks the on-disk file format (for counters)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2306) generate endpointList depending on command type (read or write)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ivan updated CASSANDRA-2306:


Attachment: rw_sep.patch

 generate endpointList depending on command type (read or write)
 ---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor
 Attachments: rw_sep.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2306) generate endpointList depending on command type (read or write)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005026#comment-13005026
 ] 

Sylvain Lebresne commented on CASSANDRA-2306:
-

What is the point of this ?

Looking (very quickly) at the patch, it seems the difference is that for read 
command, you only ever select endpoints from the the local data center. But 
this would mean that you won't ever be able to do a QUORUM or ALL unless you 
have enough replicas in the local datacenter, which is a serious drawback in 
many situation.

This is also unnecessary performance wise, because a well configured snitch 
will make sure you always choose endpoints in the local datacenter in priority 
(for reads and writes).

 generate endpointList depending on command type (read or write)
 ---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor
 Attachments: rw_sep.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Assigned: (CASSANDRA-2299) Add counters to stress tool


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne reassigned CASSANDRA-2299:
---

Assignee: Sylvain Lebresne

 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor

 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2306) generate endpointList depending on command type (read or write)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005038#comment-13005038
 ] 

ivan commented on CASSANDRA-2306:
-

bq.it seems the difference is that for read command, you only ever select 
endpoints from the the local data center. But this would mean that you won't 
ever be able to do a QUORUM or ALL unless you have enough replicas in the local 
datacenter

It's true.

bq.which is a serious drawback in many situation.

I don't think so. I think it's better to fail a read command with QUORUM or ALL 
CL if there is no enough live node in a local DC. If a read command need to get 
data from a node from another DC performance degradation will be unacceptable 
in many situation because of network latency. (This is true to write 
performance also. LOCAL_QUORUM performs much better than EACH_QUORUM.)


 generate endpointList depending on command type (read or write)
 ---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor
 Attachments: rw_sep.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2306) generate endpointList depending on command type (read or write)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005041#comment-13005041
 ] 

Sylvain Lebresne commented on CASSANDRA-2306:
-

bq. I don't think so. I think it's better to fail a read command with QUORUM or 
ALL CL if there is no enough live node in a local DC. If a read command need to 
get data from a node from another DC performance degradation will be 
unacceptable in many situation because of network latency. (This is true to 
write performance also. LOCAL_QUORUM performs much better than EACH_QUORUM.)

There are a lot of situation where people are very happy with getting that data 
even if it has to cross the atlantic than getting no data at all. And that's 
not a supposition. Hence forbidding this is a serious drawback for Cassandra.

And anyway, LOCAL_QUORUM does exactly this in a by-request configurable way, 
which is way better.  

 generate endpointList depending on command type (read or write)
 ---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor
 Attachments: rw_sep.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2294) secondary index CFs should use parent CF flush thresholds

2011-03-10 Thread Matt Conway (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005046#comment-13005046
 ] 

Matt Conway commented on CASSANDRA-2294:


Looking at jmx, the memtable settings for the index CF seem the same as for the 
parent (switch count is the only one that is different), so its not that they 
aren't getting set correctly.



 secondary index CFs should use parent CF flush thresholds
 -

 Key: CASSANDRA-2294
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2294
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 0.7.0
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.7.4

 Attachments: 2294.txt




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Resolved: (CASSANDRA-2306) generate endpointList depending on command type (read or write)

2011-03-10 Thread Brandon Williams (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams resolved CASSANDRA-2306.
-

Resolution: Not A Problem

{quote}
I think it's better to fail a read command with QUORUM or ALL CL if there is no 
enough live node in a local DC
{quote}

No, this breaks the QUORUM contract.  With an RF of 3 in 3 DCs, 5 replicas need 
to be contacted, which can exclude an entire DC.

Closing since this doesn't appear to address any kind of problem.

 generate endpointList depending on command type (read or write)
 ---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor
 Attachments: rw_sep.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Cassandra Wiki] Update of LargeDataSetConsiderations_JP by MakiWatanabe

2011-03-10 Thread Apache Wiki

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The LargeDataSetConsiderations_JP page has been changed by MakiWatanabe.
http://wiki.apache.org/cassandra/LargeDataSetConsiderations_JP?action=diffrev1=20rev2=21

--

* リペア操作にはある程度のディスク容量が必要です。（0.6では特に顕著です。0.7ではそれほどでもありません。TODO: 
具体的な最大値、依存するパラメータを明示すること。）
  
   * 
データ量が多くなるにつれ、ディスクIO操作を避けるためにキャッシュへの依存が強まります。キャバシティに関するプランニングとテストの際には以下のことを考慮すべきです。
-   * Cassandra 
の行キャッシュはJVMのヒープ上に存在し、compactionやrepairの影響を受けません。これは利点ですが、一方でメモリの有効利用という点では行キャッシュはOSのページキャッシュほど効率的でありません。
+   * Cassandra の行キャッシュはJVMのヒープ上に存在し、compactionやrepairの影響を受けません。
+   * For 0.6.8 and below, the key cache is affected by compaction because it 
is per-sstable, and compaction moves data to new sstables.
+ 
0.6.8以前はキーキャッシュはcompactionによって影響を受けます。それらのバージョンではキーキャッシュはSSTABLE単位で管理されているため、compactionによってデータが新しいSSTABLEにコピーされると古いキャッシュが無効になります。
+* Was fixed/improved as of
+ この動作は0.6.9 
及び0.7.0で改善されています。[[https://issues.apache.org/jira/browse/CASSANDRA-1878|CASSANDRA-1878]]
  
-   * For 0.6.8 and below, the key cache is affected by compaction because it 
is per-sstable, and compaction moves data to new sstables.
-* Was fixed/improved as of 
[[https://issues.apache.org/jira/browse/CASSANDRA-1878|CASSANDRA-1878]], for 
0.6.9 and 0.7.0.
-   * The operating system's page cache is affected by compaction and repair 
operations. If you are relying on the page cache to keep the active set in 
memory, you may see significant degradation on performance as a result of 
compaction and repair operations.
+ 
+   * 
OSのページキャッシュはcompaction及びrepair操作の影響を受けます。アクティブなデータをメモリ上に保つ手段としてページキャッシュに依存している場合、compaction及びrepair操作に連動して顕著な性能劣化が起きるでしょう。
+ 
+ 
-* Potential future improvements: 
[[https://issues.apache.org/jira/browse/CASSANDRA-1470|CASSANDRA-1470]], 
[[https://issues.apache.org/jira/browse/CASSANDRA-1882|CASSANDRA-1882]].
+* 
将来的な改善方法については以下のリンクで議論されています:[[https://issues.apache.org/jira/browse/CASSANDRA-1470|CASSANDRA-1470]],
 [[https://issues.apache.org/jira/browse/CASSANDRA-1882|CASSANDRA-1882]]
- * If you have column families with more than 143 million row keys in them, 
bloom filter false positive rates are likely to go up because of implementation 
concerns that limit the maximum size of a bloom filter. See 
[[ArchitectureInternals]] for information on how bloom filters are used. The 
negative effects of hitting this limit is that reads will start taking 
additional seeks to disk as the row count increases. Note that the effect you 
are seeing at any given moment will depend on when compaction was last run, 
because the bloom filter limit is per-sstable. It is an issue for column 
families because after a major compaction, the entire column family will be in 
a single sstable.
+ 
+  * bloom filterの最大サイズの実装上の制限により、14300万以上の行キーを格納しているカラムファミリでは、bloom 
filterの偽陽性率が増加することが予想されます。bloom 
filterがどのように使用されているかについては[[ArchitectureInternals]]を参照してください。この制限に抵触した場合、行数の増加に従ってreadごとに追加のseekが発生するようになります。bloom
 filterの制限はsstable単位であるため、上記の影響は最後にcompactionが実行された時間に依存することに注意してください。major 
compactionの後ではカラムファミリの全データが単一のsstableに格納されるため、これはカラムファミリ単位の問題です。
-   * This will likely be addressed in the future: See 
[[https://issues.apache.org/jira/browse/CASSANDRA-1608|CASSANDRA-1608]] and 
[[https://issues.apache.org/jira/browse/CASSANDRA-1555|CASSANDRA-1555]]
+   * 
この問題については以下のリンクで議論されています。[[https://issues.apache.org/jira/browse/CASSANDRA-1608|CASSANDRA-1608]],
 [[https://issues.apache.org/jira/browse/CASSANDRA-1555|CASSANDRA-1555]]
- * Compaction is currently not concurrent, so only a single compaction runs at 
a time. This means that sstable counts may spike during larger compactions as 
several smaller sstables are written while a large compaction is happening. 
This can cause additional seeks on reads.
+ 
+  * 
Compactionは現在は並列化されていません。即ちある瞬間に実行されるCompactionは一つのみです。これは大きなcompactionsの実行中にsstable数が増大することを意味します。大きなcompactionsの実行中は複数の小さなsstableに書き込む必要があるためです。この状態ではreadに追加のseekが必要です。
+ 
-   * Potential future improvements: 
[[https://issues.apache.org/jira/browse/CASSANDRA-1876|CASSANDRA-1876]] and 
[[https://issues.apache.org/jira/browse/CASSANDRA-1881|CASSANDRA-1881]]
+   * 
将来的な改善方法については以下のリンクで議論されています:[[https://issues.apache.org/jira/browse/CASSANDRA-1876|CASSANDRA-1876]],
 [[https://issues.apache.org/jira/browse/CASSANDRA-1881|CASSANDRA-1881]]
- * Consider the choice of file system. Removal of large files is notoriously 
slow and seek bound on e.g. ext2/ext3. Consider xfs or ext4fs. This affects 
background unlink():ing of sstables that happens every now and then, and also 
affects start-up time (if there are sstables pending removal when a node is 
starting up, they are removed as part of the start-up proceess; it may thus be 
detrimental if removing a terrabyte of sstables takes an hour (numbers are 
ballparks, not accurately measured and depends on circumstances)).
- * Adding nodes is a slow process if

[jira] Updated: (CASSANDRA-2306) generate endpointList depending on command type (read or write)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ivan updated CASSANDRA-2306:


Attachment: 2306-2.txt

 generate endpointList depending on command type (read or write)
 ---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor
 Attachments: 2306-2.txt, rw_sep.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2306) generate endpointList depending on command type (read or write)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005072#comment-13005072
 ] 

ivan commented on CASSANDRA-2306:
-

bq.No, this breaks the QUORUM contract.
Selecting endpoints just from local DC restricted to LOCAL_QUORUM CL in new 
patch.

 generate endpointList depending on command type (read or write)
 ---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor
 Attachments: 2306-2.txt, rw_sep.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Reopened: (CASSANDRA-2306) generate endpointList depending on command type (read or write)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ivan reopened CASSANDRA-2306:
-


Patch modified.

 generate endpointList depending on command type (read or write)
 ---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor
 Attachments: 2306-2.txt, rw_sep.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2299) Add counters to stress tool


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2299:


Attachment: 0001-Add-counter-support-to-stress-java.patch

Attached patch (against trunk)

This adds two new operations: COUNTER_ADD and COUNTER_GET (that really INSERT 
and READ adapted to counters).

This also add a new option: numDifferentKey. While numKey gives the number of 
insert to do, numDifferentKey tells how many different key we want. That is if 
numDifferentKey == numKey (the default), then everything acts as usual, but if 
for instance numDifferentKey == 1, then it will do numKey inserts but will 
overwrite the same row each time. This makes sense for counters where you may 
want to benchmark the reconciliation process (otherwise you don't test much 
more than what INSERT already test).

Lastly, it also add the option to disable replicate_on_write when creating the 
keyspaces.


 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8

 Attachments: 0001-Add-counter-support-to-stress-java.patch


 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2306) generate endpointList depending on command type (read or write)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005089#comment-13005089
 ] 

Jonathan Ellis commented on CASSANDRA-2306:
---

storageproxy already does what you want, by using the snitch to sort replicas 
by proximity before sending requests to the first N.

 generate endpointList depending on command type (read or write)
 ---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor
 Attachments: 2306-2.txt, rw_sep.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2299) Add counters to stress tool


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2299:


Remaining Estimate: 4h  (was: 0h)

 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8

 Attachments: 0001-Add-counter-support-to-stress-java.patch

  Time Spent: 4h
  Remaining Estimate: 4h

 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Resolved: (CASSANDRA-2306) generate endpointList depending on command type (read or write)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis resolved CASSANDRA-2306.
---

Resolution: Invalid

 generate endpointList depending on command type (read or write)
 ---

 Key: CASSANDRA-2306
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2306
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: ivan
Priority: Minor
 Attachments: 2306-2.txt, rw_sep.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2299) Add counters to stress tool


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2299:


Remaining Estimate: 0h  (was: 4h)

 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8

 Attachments: 0001-Add-counter-support-to-stress-java.patch

  Time Spent: 4h
  Remaining Estimate: 0h

 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-1924) Broken keyspace strategy_option with zero replicas


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-1924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-1924:
--

Affects Version/s: (was: 0.7.0 rc 3)
   (was: 0.7.0 rc 2)
   (was: 0.7.0 rc 1)
   (was: 0.7 beta 2)
   (was: 0.7 beta 3)
   (was: 0.7 beta 1)
   0.7.0
Fix Version/s: 0.7.4
 Assignee: Sylvain Lebresne

 Broken keyspace strategy_option with zero replicas
 --

 Key: CASSANDRA-1924
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1924
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.7.0
Reporter: Thor Carpenter
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.7.4


 When a keyspace is defined that has strategy options specifying zero replicas 
 should be place in a datacenter (e.g. DCX:0), an assert is violated for any 
 insert and LOCAL_QUORUM reads fail.  I'm not sure if the issue is that there 
 are no nodes in DCX or that I'm saying DCX shouldn't get any replicas, or a 
 combination of the two.
 The broken keyspace:
 create keyspace KeyspaceDC1 with
   replication_factor = 1 and
   placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' 
 and
   strategy_options = [{DC1:1, DC2:0}];
 The fixed keyspace:
 create keyspace KeyspaceDC1 with
   replication_factor = 1 and
   placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' 
 and
   strategy_options = [{DC1:1}];
 To reproduce:
 * Install the 0.7rc3 rpm on a single node in DC1.
 * In cassandra.yaml set initial_token = 1 and specify PropertyFileSnitch.
 * cassandra-topology.properties:
 10.5.64.26=DC1:R1
 default=DC2:R1
 * Schema loaded via cassandra-cli:
 create keyspace KeyspaceDC1 with
   replication_factor = 1 and
   placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' 
 and
   strategy_options = [{DC1:1, DC2:0}];
 use KeyspaceDC1;
 create column family TestCF with
   column_type = 'Standard' and
   comparator = 'BytesType' and
   keys_cached = 20 and
   rows_cached = 2000 and
   gc_grace = 0 and
   read_repair_chance = 0.0;
 * In cassandra-cli execute the following:
 [default@unknown] use KeyspaceDC1;
 Authenticated to keyspace: KeyspaceDC1
 [default@KeyspaceDC1] set TestCF['some key']['some col'] = 'some value';
 Internal error processing insert
 * If you have asserts enabled, check system.log where you should find the 
 assertion error: 
 DEBUG [pool-1-thread-3] 2010-12-29 12:10:38,897 CassandraServer.java (line 
 362) insert
 ERROR [pool-1-thread-3] 2010-12-29 12:10:38,906 Cassandra.java (line 2960) 
 Internal error processing insert
 java.lang.AssertionError
 at 
 org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:392)
  
 at 
 org.apache.cassandra.locator.TokenMetadata.ringIterator(TokenMetadata.java:417)
 at 
 org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:95)
 at 
 org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:99)
 at 
 org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1411)
 at 
 org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1394)
 at 
 org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:109)
 at 
 org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:442)
 at 
 org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:379)
 at 
 org.apache.cassandra.thrift.Cassandra$Processor$insert.process(Cassandra.java:2952)
 at 
 org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555)
 at 
 org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  
 at java.lang.Thread.run(Thread.java:619)
 * If you don't have asserts enabled, you should find that no errors are 
 logged but LOCAL_QUORUM reads cause TimedOutExceptions on the client.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (CASSANDRA-2307) IndexOutOfBoundsException during compaction

IndexOutOfBoundsException during compaction
---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.3, 0.7.2
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau


We're getting an IndexOutOfBounds exception when compacting.

Here's the detailed error we get on-screen when running nodetool -h 
10.3.133.10 compact:

Error occured while compacting keyspace test
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at 
org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
at 
org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
at 
org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
at 
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
Source)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
Source)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Unknown Source)
at java.nio.HeapByteBuffer.getInt(Unknown Source)
at 
org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
at 
org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
at 
org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
at 
org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
... 3 more

And here's the error I'm getting in my log file:

ERROR [CompactionExecutor:1] 2011-03-09 19:16:52,299 
AbstractCassandraDaemon.java (line 114) Fatal exception in thread 
Thread[CompactionExecutor:1,1,main]
java.lang.IndexOutOfBoundsException
at

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction

2011-03-10 Thread JIRA


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005097#comment-13005097
 ] 

Sébastien Giroux commented on CASSANDRA-2307:
-

Did you try running nodetool -h localhost scrub on that node ? I'm not sure 
if you're running into data corruption but I remember having a similar problem 
before running scrub, maybe it's something else tho, but worth a try I suppose!

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at 
 org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
 at

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005106#comment-13005106
 ] 

Mike Nadeau commented on CASSANDRA-2307:


I ran into this when running the scrub :

java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap 
space

I'm increasing the heap to 10G (it was 4G) and I rerun it right now.

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at 
 org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
 at 
 org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
 at

svn commit: r1080254 - in /cassandra/trunk: src/java/org/apache/cassandra/cql/Cql.g src/java/org/apache/cassandra/cql/Term.java test/system/test_cql.py

Author: eevans
Date: Thu Mar 10 15:45:59 2011
New Revision: 1080254

URL: http://svn.apache.org/viewvc?rev=1080254view=rev
Log:
remove unicode term type

Patch by eevans for CASSANDRA-2027

Modified:
cassandra/trunk/src/java/org/apache/cassandra/cql/Cql.g
cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java
cassandra/trunk/test/system/test_cql.py

Modified: cassandra/trunk/src/java/org/apache/cassandra/cql/Cql.g
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/cql/Cql.g?rev=1080254r1=1080253r2=1080254view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/cql/Cql.g (original)
+++ cassandra/trunk/src/java/org/apache/cassandra/cql/Cql.g Thu Mar 10 15:45:59 
2011
@@ -270,7 +270,7 @@ term returns [Term item]
 ;
 
 literal returns [Term term]
-: ( t=STRING_LITERAL | t=INTEGER | t=UNICODE | t=UUID ) { $term = new 
Term($t.text, $t.type); }
+: ( t=STRING_LITERAL | t=INTEGER | t=UUID ) { $term = new Term($t.text, 
$t.type); }
 ;
 
 termList returns [ListTerm items]
@@ -421,11 +421,7 @@ IDENT
 COMPIDENT
 : IDENT ( ':' IDENT)*
 ;
-
-UNICODE
-: 'u' STRING_LITERAL
-;
-
+   
 UUID
 : HEX HEX HEX HEX HEX HEX HEX HEX '-'
   HEX HEX HEX HEX '-'

Modified: cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java?rev=1080254r1=1080253r2=1080254view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java (original)
+++ cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java Thu Mar 10 
15:45:59 2011
@@ -124,8 +124,6 @@ public class Term
 return AsciiType.instance.fromString(text);
 case INTEGER: 
 return IntegerType.instance.fromString(text);
-case UNICODE:
-return UTF8Type.instance.fromString(text);
 case UUID:
 return LexicalUUIDType.instance.fromString(text);
 case TIMEUUID:
@@ -155,7 +153,7 @@ public class Term
 
 enum TermType
 {
-STRING, INTEGER, UNICODE, UUID, TIMEUUID;
+STRING, INTEGER, UUID, TIMEUUID;
 
 static TermType forInt(int type)
 {
@@ -163,8 +161,6 @@ enum TermType
 return STRING;
 else if (type == CqlParser.INTEGER)
 return INTEGER;
-else if (type == CqlParser.UNICODE)
-return UNICODE;
 else if (type == CqlParser.UUID)
 return UUID;
 

Modified: cassandra/trunk/test/system/test_cql.py
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/test/system/test_cql.py?rev=1080254r1=1080253r2=1080254view=diff
==
--- cassandra/trunk/test/system/test_cql.py (original)
+++ cassandra/trunk/test/system/test_cql.py Thu Mar 10 15:45:59 2011
@@ -61,10 +61,10 @@ def load_sample(dbconn):
 dbconn.execute(query, cd1, vd1, col, val, kd)
 
 dbconn.execute(
-UPDATE StandardUtf82 SET u'%s' = 've1', 'col' = 'val' WHERE KEY = 'kd'
+UPDATE StandardUtf82 SET '%s' = 've1', 'col' = 'val' WHERE KEY = 'kd'
  % u'\xa9'.encode('utf8'))
 dbconn.execute(
-UPDATE StandardUtf82 SET u'cf1' = 'vf1', 'col' = 'val' WHERE KEY = 'kd'
+UPDATE StandardUtf82 SET 'cf1' = 'vf1', 'col' = 'val' WHERE KEY = 'kd'
 )
 
 dbconn.execute(

svn commit: r1080255 - /cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java

Author: eevans
Date: Thu Mar 10 15:46:14 2011
New Revision: 1080255

URL: http://svn.apache.org/viewvc?rev=1080255view=rev
Log:
cleanup cruft left behind after refactorings

Patch by eevans for CASSANDRA-2027

Modified:
cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java

Modified: cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java?rev=1080255r1=1080254r2=1080255view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java (original)
+++ cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java Thu Mar 10 
15:46:14 2011
@@ -20,9 +20,7 @@
  */
 package org.apache.cassandra.cql;
 
-import java.io.UnsupportedEncodingException;
 import java.nio.ByteBuffer;
-import java.text.ParseException;
 
 import org.apache.cassandra.db.marshal.AbstractType;
 import org.apache.cassandra.db.marshal.AsciiType;
@@ -30,29 +28,11 @@ import org.apache.cassandra.db.marshal.I
 import org.apache.cassandra.db.marshal.LexicalUUIDType;
 import org.apache.cassandra.db.marshal.MarshalException;
 import org.apache.cassandra.db.marshal.TimeUUIDType;
-import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.thrift.InvalidRequestException;
-import org.apache.cassandra.utils.ByteBufferUtil;
-import org.apache.cassandra.utils.FBUtilities;
-import org.apache.cassandra.utils.UUIDGen;
-import org.apache.commons.lang.time.DateUtils;
 
 /** A term parsed from a CQL statement. */
 public class Term
-{
-private static String[] iso8601Patterns = new String[] {
--MM-dd HH:mm,
--MM-dd HH:mm:ss,
--MM-dd HH:mmZ,
--MM-dd HH:mm:ssZ,
--MM-dd'T'HH:mm,
--MM-dd'T'HH:mmZ,
--MM-dd'T'HH:mm:ss,
--MM-dd'T'HH:mm:ssZ,
--MM-dd,
--MM-ddZ
-};
-
+{   
 private final String text;
 private final TermType type;

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005115#comment-13005115
 ] 

Mike Nadeau commented on CASSANDRA-2307:


OK the scrub was successful with 10G of heap. Now retrying the compaction. It's 
the first time I run a scrub, what is it exactly?

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at 
 org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
 at 
 org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
 at 
 org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005119#comment-13005119
 ] 

Mike Nadeau commented on CASSANDRA-2307:


No luck :( I'm still getting this error during the compaction:

Error occured during compaction
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at 
org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:209)
at 
org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1720)
at 
org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1263)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown 
Source)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown 
Source)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
at 
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
Source)
at 
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
Source)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Unknown Source)
at java.nio.HeapByteBuffer.getInt(Unknown Source)
at 
org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
at 
org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:852)
at 
org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:839)
at 
org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:830)
at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
at 
org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:240)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
... 3 more

I'll be happy to provide SSTables, I just don't know which files would be 
helpful.

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project:

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005131#comment-13005131
 ] 

Sylvain Lebresne commented on CASSANDRA-2307:
-

Check the log. Just before the exceptions, you should see a message saying 
'Compacting [...]' with a number of -Data.db files. Those are the useful ones. 
It would be best if you also join with those the filter, index and statistics 
files (i.e, files with the same number but ending in -Filter.db, -Index.db and 
-Statistics.db instead of -Data.db)

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at

[jira] Updated: (CASSANDRA-2305) Tombstoned rows not purged after gcgraceseconds


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2305:
--

 Priority: Minor  (was: Major)
Fix Version/s: 0.7.4
 Assignee: Sylvain Lebresne
  Summary: Tombstoned rows not purged after gcgraceseconds  (was: 
Tombstoned CFs not reset after compaction)

 Tombstoned rows not purged after gcgraceseconds
 ---

 Key: CASSANDRA-2305
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.7.0
Reporter: Jeffrey Wang
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.7.4


 From email to list:
 I was wondering if this is the expected behavior of deletes (0.7.0). Let's 
 say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. 
 The following sequence of operations happens (in the given order):
 insert row X with timestamp T
 delete row X with timestamp T+1
 force flush + compaction
 insert row X with timestamp T
 My understanding is that the tombstone created by the delete (and row X) will 
 disappear with the flush + compaction which means the last insertion should 
 show up. My experimentation, however, suggests otherwise (the last insertion 
 does not show up).
 I believe I have traced this to the fact that the markedForDeleteAt field on 
 the ColumnFamily does not get reset after a compaction (after 
 gc_grace_seconds has passed); is this desirable? I think it introduces an 
 inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2304) sstable2json dies with Too many open files, regardless of ulimit


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2304:
--

Attachment: 2304.txt

patch to close column iterator and only do one pass per row

 sstable2json dies with Too many open files, regardless of ulimit
 --

 Key: CASSANDRA-2304
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2304
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Affects Versions: 0.7.1
Reporter: Jason Harvey
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.7.4

 Attachments: 2304.txt, sstable.tar.bz2


 Running sstable2json on the attached sstable eventually results in the 
 following:
 {code}
 Exception in thread main java.io.IOError: java.io.FileNotFoundException: 
 /var/lib/cassandra/data/reddit/CommentSortsCache-f-9764-Data.db (Too many 
 open files)
 at 
 org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:68)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:567)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:49)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:68)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
 at 
 org.apache.cassandra.tools.SSTableExport.serializeRow(SSTableExport.java:187)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:355)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:377)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:390)
 at 
 org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:448)
 Caused by: java.io.FileNotFoundException: 
 /var/lib/cassandra/data/reddit/CommentSortsCache-f-9764-Data.db (Too many 
 open files)
 at java.io.RandomAccessFile.open(Native Method)
 at java.io.RandomAccessFile.init(RandomAccessFile.java:233)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:111)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:106)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:91)
 at 
 org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:62)
 {code}
 Set my ulimit -n to 6 and got the same result. Leaking file descriptors?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2304) sstable2json dies with Too many open files, regardless of ulimit


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2304:
--

 Reviewer: xedin
  Component/s: Tools
 Priority: Minor  (was: Major)
Affects Version/s: 0.7.1
Fix Version/s: 0.7.4
 Assignee: Jonathan Ellis

This is a bug with non-mmap'd I/O.

 sstable2json dies with Too many open files, regardless of ulimit
 --

 Key: CASSANDRA-2304
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2304
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Affects Versions: 0.7.1
Reporter: Jason Harvey
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.7.4

 Attachments: 2304.txt, sstable.tar.bz2


 Running sstable2json on the attached sstable eventually results in the 
 following:
 {code}
 Exception in thread main java.io.IOError: java.io.FileNotFoundException: 
 /var/lib/cassandra/data/reddit/CommentSortsCache-f-9764-Data.db (Too many 
 open files)
 at 
 org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:68)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:567)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:49)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:68)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
 at 
 org.apache.cassandra.tools.SSTableExport.serializeRow(SSTableExport.java:187)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:355)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:377)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:390)
 at 
 org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:448)
 Caused by: java.io.FileNotFoundException: 
 /var/lib/cassandra/data/reddit/CommentSortsCache-f-9764-Data.db (Too many 
 open files)
 at java.io.RandomAccessFile.open(Native Method)
 at java.io.RandomAccessFile.init(RandomAccessFile.java:233)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:111)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:106)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:91)
 at 
 org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:62)
 {code}
 Set my ulimit -n to 6 and got the same result. Leaking file descriptors?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2284) Make changes to the set of memtables and sstables of a cfstore atomic

[
https://issues.apache.org/jira/browse/CASSANDRA-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005147#comment-13005147
]

Sylvain Lebresne commented on CASSANDRA-2284:
-

Truth is that, the sheer size of CFStore (over 2K line) annoys me. I think this
prove CFStore has too much things in it and we messed up at encapsulation. So I
tried to make DataTracker be about dealing with the memtables and sstables as
much as possible. And being able to have View not leak from DataTracker sound
like a proof that we do have some encapsulation there. I'm not claiming it's
perfect, but I really don't think that CFStore is our most beautiful class in
there and I do believe that DataTracker improves this a bit.

But there is a good part of subjectivity in all this anyway. And if people
really don't like the move of the bits of querying in there, fine, I'd move
them back in CFStore and leak the View from DataTracker. I would however be
more reluctant to remove DataTracker altogether and put the AtomicReference in
CFStore. Because this would move all the CAS related logic in CFStore and
seriously there is already too much stuff in CFStore. My humble opinion though.

Make changes to the set of memtables and sstables of a cfstore atomic
--

Key: CASSANDRA-2284
URL: https://issues.apache.org/jira/browse/CASSANDRA-2284
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
Fix For: 0.8

Attachments: 0001-Make-memtable-and-sstable-switches-atomic.patch

Original Estimate: 8h
Remaining Estimate: 8h

Some switches happen in the set of memtables and sstables of a cfstore:
* when we switch the memtable, the current memtable is replaced by a new
one and add to the memtablesPendingFlush
* when a memtable is fully flushed, it is removed from
memtablesPendingFlush and the newly created sstable is added to the set of
active sstables.
* after compaction, compacted memtables are removed from the active
sstables and the compacted sstable is added instead
Only the last of these operations is atomic. This ticket proposes to makes
all of them atomic, using the idea of the View idea hinted by Stu in the
comments of CASSANDRA-1954.
The main reason for this is to fix CASSANDRA-2105. But I think that another
benefit of this is to make reasoning about those operations easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2304) sstable2json dies with Too many open files, regardless of ulimit

2011-03-10 Thread Pavel Yaskevich (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Yaskevich updated CASSANDRA-2304:
---

Attachment: CASSANDRA-2304-v2.patch

with number of the exported columns properly incremented. LGTM.

 sstable2json dies with Too many open files, regardless of ulimit
 --

 Key: CASSANDRA-2304
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2304
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Affects Versions: 0.7.1
Reporter: Jason Harvey
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.7.4

 Attachments: 2304.txt, CASSANDRA-2304-v2.patch, sstable.tar.bz2


 Running sstable2json on the attached sstable eventually results in the 
 following:
 {code}
 Exception in thread main java.io.IOError: java.io.FileNotFoundException: 
 /var/lib/cassandra/data/reddit/CommentSortsCache-f-9764-Data.db (Too many 
 open files)
 at 
 org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:68)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:567)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:49)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:68)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
 at 
 org.apache.cassandra.tools.SSTableExport.serializeRow(SSTableExport.java:187)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:355)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:377)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:390)
 at 
 org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:448)
 Caused by: java.io.FileNotFoundException: 
 /var/lib/cassandra/data/reddit/CommentSortsCache-f-9764-Data.db (Too many 
 open files)
 at java.io.RandomAccessFile.open(Native Method)
 at java.io.RandomAccessFile.init(RandomAccessFile.java:233)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:111)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:106)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:91)
 at 
 org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:62)
 {code}
 Set my ulimit -n to 6 and got the same result. Leaking file descriptors?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2299) Add counters to stress tool


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005161#comment-13005161
 ] 

T Jake Luciani commented on CASSANDRA-2299:
---

Oh I had already done this... but yours has more options i see.

 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8

 Attachments: 0001-Add-counter-support-to-stress-java.patch

  Time Spent: 4h
  Remaining Estimate: 0h

 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (CASSANDRA-2308) track row indexes in sstable stats

track row indexes in sstable stats
--

 Key: CASSANDRA-2308
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2308
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Priority: Minor
 Fix For: 0.8


Exposing row index sizes would give us the information to tune 
column_index_size correctly (see CASSANDRA-2297).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2299) Add counters to stress tool


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-2299:
--

Attachment: 0001-CASSANDRA-2299-Alternate-counter-stress-impl.txt

 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8

 Attachments: 0001-Add-counter-support-to-stress-java.patch, 
 0001-CASSANDRA-2299-Alternate-counter-stress-impl.txt

  Time Spent: 4h
  Remaining Estimate: 0h

 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2297) UnsupportedOperationException: Overflow in bytesPastMark(..)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2297:
--

Attachment: 2297.txt

fix bytesPastMark to return long.

Note that if your row index is  2GB then you should probably increase 
column_index_size_in_kb to 256 or higher.  I've created CASSANDRA-2308 to 
expose row index size so this can be tuned more accurately.

 UnsupportedOperationException: Overflow in bytesPastMark(..)
 

 Key: CASSANDRA-2297
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2297
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.7.0
 Environment: Java 1.6.0_23, CentOS 5.5 (64bit)
Reporter: Muga Nishizawa
 Fix For: 0.7.4

 Attachments: 2297.txt


 I hit the following exception on a row that was more than 60GB.  
 The row has column families of super column type.
 This problem is discussed by the following thread.  
 http://www.mail-archive.com/dev@cassandra.apache.org/msg01881.html
 {code}
 ERROR [HintedHandoff:1] 2011-02-26 18:49:35,708 
 DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor
 java.lang.RuntimeException: java.lang.UnsupportedOperationException: 
 Overflow: 2147484294
 at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.UnsupportedOperationException: Overflow: 2147484294
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.bytesPastMark(BufferedRandomAccessFile.java:477)
 at 
 org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:179)
 at 
 org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:120)
 at 
 org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:1)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
 at 
 org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
 at 
 org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
 at 
 org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:118)
 at 
 org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:142)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1290)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1167)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1095)
 at 
 org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:138)
 at 
 org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:313)
 at 
 org.apache.cassandra.db.HintedHandOffManager.access$1(HintedHandOffManager.java:262)
 at 
 org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:391)
 at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
 ... 3 more
 {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2299) Add counters to stress tool


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005165#comment-13005165
 ] 

Sylvain Lebresne commented on CASSANDRA-2299:
-

Oups. You should have assigned it to yourself :)

 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8

 Attachments: 0001-Add-counter-support-to-stress-java.patch, 
 0001-CASSANDRA-2299-Alternate-counter-stress-impl.txt

  Time Spent: 4h
  Remaining Estimate: 0h

 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2288) AES Counter Repair Improvements


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2288:
--

 Reviewer: slebresne
 Priority: Minor  (was: Major)
Fix Version/s: 0.8

 AES Counter Repair Improvements
 ---

 Key: CASSANDRA-2288
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2288
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 0.8
Reporter: Alan Liang
Assignee: Alan Liang
Priority: Minor
 Fix For: 0.8

 Attachments: CASSANDRA-2288-aes_counter_repair_improvements.diff


 A few issues found for AES Counter Repair in 
 AESCommutativeRowIndexer#doIndexing:
 - sync() being called for each row in sstable
 - because the sstable is rebuilt inline (read and write on same file), this 
 causes seeking back and forth of write and read positions which causes many 
 flushes
 - BufferedRandomAccessFile#setLength does not work with buffers
 Fixed:
 - remove sync() until end
 - use two BufferedRandomAccessFile's one for reader, one for writer
 - cache length of reader file
 - implement BufferedRandomAccessFile#setLength to work with buffer

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1080282 - in /cassandra/trunk: src/java/org/apache/cassandra/cql/Cql.g src/java/org/apache/cassandra/cql/Term.java test/system/test_cql.py

Author: eevans
Date: Thu Mar 10 17:11:18 2011
New Revision: 1080282

URL: http://svn.apache.org/viewvc?rev=1080282view=rev
Log:
remove timeuuid term type

Patch by eevans for CASSANDRA-2027

Modified:
cassandra/trunk/src/java/org/apache/cassandra/cql/Cql.g
cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java
cassandra/trunk/test/system/test_cql.py

Modified: cassandra/trunk/src/java/org/apache/cassandra/cql/Cql.g
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/cql/Cql.g?rev=1080282r1=1080281r2=1080282view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/cql/Cql.g (original)
+++ cassandra/trunk/src/java/org/apache/cassandra/cql/Cql.g Thu Mar 10 17:11:18 
2011
@@ -266,11 +266,7 @@ comparatorType
 ;
 
 term returns [Term item]
-: ( t=timeuuid | t=literal ) { $item = t; }
-;
-
-literal returns [Term term]
-: ( t=STRING_LITERAL | t=INTEGER | t=UUID ) { $term = new Term($t.text, 
$t.type); }
+: ( t=STRING_LITERAL | t=INTEGER | t=UUID ) { $item = new Term($t.text, 
$t.type); }
 ;
 
 termList returns [ListTerm items]
@@ -298,10 +294,6 @@ truncateStatement returns [String cfam]
 endStmnt
 : (EOF | ';')
 ;
-
-timeuuid returns [Term term]
-: 'timeuuid(' uuidstr=( STRING_LITERAL | INTEGER )? ')' { return new 
Term($uuidstr.text, TermType.TIMEUUID); }
-;
 
 
 // Case-insensitive keywords

Modified: cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java?rev=1080282r1=1080281r2=1080282view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java (original)
+++ cassandra/trunk/src/java/org/apache/cassandra/cql/Term.java Thu Mar 10 
17:11:18 2011
@@ -27,7 +27,6 @@ import org.apache.cassandra.db.marshal.A
 import org.apache.cassandra.db.marshal.IntegerType;
 import org.apache.cassandra.db.marshal.LexicalUUIDType;
 import org.apache.cassandra.db.marshal.MarshalException;
-import org.apache.cassandra.db.marshal.TimeUUIDType;
 import org.apache.cassandra.thrift.InvalidRequestException;
 
 /** A term parsed from a CQL statement. */
@@ -106,8 +105,6 @@ public class Term
 return IntegerType.instance.fromString(text);
 case UUID:
 return LexicalUUIDType.instance.fromString(text);
-case TIMEUUID:
-return TimeUUIDType.instance.fromString(text);
 }
 
 // FIXME: handle scenario that should never happen
@@ -133,7 +130,7 @@ public class Term
 
 enum TermType
 {
-STRING, INTEGER, UUID, TIMEUUID;
+STRING, INTEGER, UUID;
 
 static TermType forInt(int type)
 {

Modified: cassandra/trunk/test/system/test_cql.py
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/test/system/test_cql.py?rev=1080282r1=1080281r2=1080282view=diff
==
--- cassandra/trunk/test/system/test_cql.py (original)
+++ cassandra/trunk/test/system/test_cql.py Thu Mar 10 17:11:18 2011
@@ -439,20 +439,18 @@ class TestCql(ThriftTester):
 # Store and retrieve a timeuuid using it's hex-formatted string
 timeuuid = uuid.uuid1()
 conn.execute(
-UPDATE StandardTimeUUID SET timeuuid('%s') = 10
-WHERE KEY = 'uuidtest'
+UPDATE StandardTimeUUID SET '%s' = 10 WHERE KEY = 'uuidtest'
  % str(timeuuid))
 
 r = conn.execute(
-SELECT timeuuid('%s') FROM StandardTimeUUID WHERE KEY = 'uuidtest'
+SELECT '%s' FROM StandardTimeUUID WHERE KEY = 'uuidtest'
  % str(timeuuid))
 assert r[0].columns[0].name == timeuuid.bytes
 
 # Tests a node-side conversion from long to UUID.
 ms = uuid1bytes_to_millis(uuid.uuid1().bytes)
 conn.execute(
-UPDATE StandardTimeUUIDValues SET 'id' = timeuuid(%d)
-WHERE KEY = 'uuidtest'
+UPDATE StandardTimeUUIDValues SET 'id' = %d WHERE KEY = 'uuidtest'
  % ms)
 
 r = conn.execute(
@@ -462,8 +460,7 @@ class TestCql(ThriftTester):
 
 # Tests a node-side conversion from ISO8601 to UUID.
 conn.execute(
-UPDATE StandardTimeUUIDValues
-SET 'id2' = timeuuid('2011-01-31 17:00:00-') 
+UPDATE StandardTimeUUIDValues SET 'id2' = '2011-01-31 
17:00:00-'
 WHERE KEY = 'uuidtest'
 )
 
@@ -475,31 +472,18 @@ class TestCql(ThriftTester):
 assert ms == 129649320, \
 %d != 129649320 (2011-01-31 17:00:00-) % ms
 
-# Tests node-side conversion of empty term to UUID
-conn.execute(
-UPDATE StandardTimeUUIDValues SET 'id3' = timeuuid()
-WHERE KEY

svn commit: r1080283 - /cassandra/trunk/src/java/org/apache/cassandra/db/marshal/TimeUUIDType.java

Author: eevans
Date: Thu Mar 10 17:11:24 2011
New Revision: 1080283

URL: http://svn.apache.org/viewvc?rev=1080283view=rev
Log:
empty term no longer makes sense for TimeUUID

Patch by eevans for CASSANDRA-2027

Modified:
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/TimeUUIDType.java

Modified: 
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/TimeUUIDType.java
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/marshal/TimeUUIDType.java?rev=1080283r1=1080282r2=1080283view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/db/marshal/TimeUUIDType.java 
(original)
+++ cassandra/trunk/src/java/org/apache/cassandra/db/marshal/TimeUUIDType.java 
Thu Mar 10 17:11:24 2011
@@ -139,7 +139,7 @@ public class TimeUUIDType extends Abstra
 if (uuid.version() != 1)
 throw new MarshalException(TimeUUID supports only version 1 
UUIDs);
 }
-else if (source.equals() || source.toLowerCase().equals(now))
+else if (source.toLowerCase().equals(now))
 {
 idBytes = 
ByteBuffer.wrap(UUIDGen.decompose(UUIDGen.makeType1UUIDFromHost(FBUtilities.getLocalAddress(;
 }

svn commit: r1080286 - in /cassandra/branches/cassandra-0.7: CHANGES.txt src/java/org/apache/cassandra/tools/SSTableExport.java

Author: jbellis
Date: Thu Mar 10 17:13:16 2011
New Revision: 1080286

URL: http://svn.apache.org/viewvc?rev=1080286view=rev
Log:
fix fd leak in sstable2json with non-mmap'd i/o
patch by jbellis; reviewed by Pavel Yaskevich for CASSANDRA-2304

Modified:
cassandra/branches/cassandra-0.7/CHANGES.txt

cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/tools/SSTableExport.java

Modified: cassandra/branches/cassandra-0.7/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/CHANGES.txt?rev=1080286r1=1080285r2=1080286view=diff
==
--- cassandra/branches/cassandra-0.7/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.7/CHANGES.txt Thu Mar 10 17:13:16 2011
@@ -18,6 +18,7 @@
(CASSANDRA-2282)
  * fix commitlog replay when flush position refers to data that didn't
get synced before server died (CASSANDRA-2285)
+ * fix fd leak in sstable2json with non-mmap'd i/o (CASSANDRA-2304)
 
 
 0.7.3

Modified: 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/tools/SSTableExport.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/tools/SSTableExport.java?rev=1080286r1=1080285r2=1080286view=diff
==
--- 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/tools/SSTableExport.java
 (original)
+++ 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/tools/SSTableExport.java
 Thu Mar 10 17:13:16 2011
@@ -19,6 +19,7 @@
 package org.apache.cassandra.tools;
 
 import java.io.File;
+import java.io.IOError;
 import java.io.IOException;
 import java.io.PrintStream;
 import java.nio.ByteBuffer;
@@ -37,6 +38,7 @@ import org.apache.cassandra.config.Confi
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.dht.IPartitioner;
 import org.apache.cassandra.io.sstable.*;
+import org.apache.cassandra.utils.Pair;
 
 import static org.apache.cassandra.utils.ByteBufferUtil.bytesToHex;
 import static org.apache.cassandra.utils.ByteBufferUtil.hexToBytes;
@@ -97,16 +99,23 @@ public class SSTableExport
  * Serialize columns using given column iterator
  * @param columns column iterator
  * @param out output stream
+ * @return pair of (number of columns serialized, last column serialized)
  */
-private static void serializeColumns(IteratorIColumn columns, 
PrintStream out)
+private static PairInteger, ByteBuffer 
serializeColumns(IteratorIColumn columns, PrintStream out)
 {
+int n = 0;
+IColumn column = null;
 while (columns.hasNext())
 {
-serializeColumn(columns.next(), out);
+column = columns.next();
+n++;
+serializeColumn(column, out);
 
 if (columns.hasNext())
 out.print(, );
 }
+
+return new PairInteger, ByteBuffer(n, column == null ? null : 
column.name());
 }
 
 /**
@@ -174,25 +183,29 @@ public class SSTableExport
 
 IColumnIterator columns = filter.getSSTableColumnIterator(reader);
 
-int columnCount = 0;
-while (columns.hasNext())
-{
-// setting new start column to the last of the current columns
-startColumn = columns.next().name();
-columnCount++;
-}
-
+PairInteger, ByteBuffer serialized;
 try
 {
-columns = filter.getSSTableColumnIterator(reader); // iterator 
reset
-serializeRow(columns, isSuperCF, out);
+serialized = serializeRow(columns, isSuperCF, out);
 }
 catch (IOException e)
 {
 System.err.println(WARNING: Corrupt row  + key +  
(skipping).);
+continue;
+}
+finally
+{
+try
+{
+columns.close();
+}
+catch (IOException e)
+{
+throw new IOError(e);
+}
 }
 
-if (columnCount  PAGE_SIZE)
+if (serialized.left  PAGE_SIZE)
 break;
 
 out.print(,);
@@ -207,16 +220,20 @@ public class SSTableExport
  * @param columns columns of the row
  * @param isSuper true if wrapping Column Family is Super
  * @param out output stream
+ * @return pair of (number of columns serialized, last column serialized)
  *
  * @throws IOException on any I/O error.
  */
-private static void serializeRow(IColumnIterator columns, boolean isSuper, 
PrintStream out) throws IOException
+private static PairInteger, ByteBuffer serializeRow(IColumnIterator 
columns, boolean isSuper, PrintStream out) throws IOException
 {
 if (isSuper)
 {
+int

[jira] Commented: (CASSANDRA-2299) Add counters to stress tool

2011-03-10 Thread Pavel Yaskevich (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005187#comment-13005187
 ] 

Pavel Yaskevich commented on CASSANDRA-2299:


Gotcha, Only one thing left - please add my additions to your previous patch 
and attach version 2 and everything is ready to be committed.

 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8

 Attachments: 0001-Add-counter-support-to-stress-java.patch, 
 0001-CASSANDRA-2299-Alternate-counter-stress-impl.txt, help-message-fix.patch

  Time Spent: 4h
  Remaining Estimate: 0h

 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2299) Add counters to stress tool


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2299:


Attachment: 0001-Add-counter-support-to-stress-java-v2.patch

Attaching v2 with the first help typo fix. I did not include the second one 
because no-replicate-on-write is an option with no argument (it's a flag). So 
to indicate that the default is true feel weird. Even more weird since the 
option is called no-replicate-on-write while the default is to replicate on 
write.

 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8

 Attachments: 0001-Add-counter-support-to-stress-java-v2.patch, 
 0001-Add-counter-support-to-stress-java.patch, 
 0001-CASSANDRA-2299-Alternate-counter-stress-impl.txt, help-message-fix.patch

  Time Spent: 4h
  Remaining Estimate: 0h

 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2299) Add counters to stress tool

2011-03-10 Thread Pavel Yaskevich (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005194#comment-13005194
 ] 

Pavel Yaskevich commented on CASSANDRA-2299:


+1

 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8

 Attachments: 0001-Add-counter-support-to-stress-java-v2.patch, 
 0001-Add-counter-support-to-stress-java.patch, 
 0001-CASSANDRA-2299-Alternate-counter-stress-impl.txt, help-message-fix.patch

  Time Spent: 4h
  Remaining Estimate: 0h

 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2304) sstable2json dies with Too many open files, regardless of ulimit


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005198#comment-13005198
 ] 

Hudson commented on CASSANDRA-2304:
---

Integrated in Cassandra-0.7 #370 (See 
[https://hudson.apache.org/hudson/job/Cassandra-0.7/370/])
fix fd leak in sstable2json with non-mmap'd i/o
patch by jbellis; reviewed by Pavel Yaskevich for CASSANDRA-2304


 sstable2json dies with Too many open files, regardless of ulimit
 --

 Key: CASSANDRA-2304
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2304
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Affects Versions: 0.7.1
Reporter: Jason Harvey
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.7.4

 Attachments: 2304.txt, CASSANDRA-2304-v2.patch, sstable.tar.bz2


 Running sstable2json on the attached sstable eventually results in the 
 following:
 {code}
 Exception in thread main java.io.IOError: java.io.FileNotFoundException: 
 /var/lib/cassandra/data/reddit/CommentSortsCache-f-9764-Data.db (Too many 
 open files)
 at 
 org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:68)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:567)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:49)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:68)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
 at 
 org.apache.cassandra.tools.SSTableExport.serializeRow(SSTableExport.java:187)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:355)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:377)
 at 
 org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:390)
 at 
 org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:448)
 Caused by: java.io.FileNotFoundException: 
 /var/lib/cassandra/data/reddit/CommentSortsCache-f-9764-Data.db (Too many 
 open files)
 at java.io.RandomAccessFile.open(Native Method)
 at java.io.RandomAccessFile.init(RandomAccessFile.java:233)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:111)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:106)
 at 
 org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:91)
 at 
 org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:62)
 {code}
 Set my ulimit -n to 6 and got the same result. Leaking file descriptors?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005197#comment-13005197
 ] 

Sylvain Lebresne commented on CASSANDRA-2307:
-

bq. Do someone know if we can always safely remove the snapshots folder? I 
think it was created by my nodetool repair command (which I interrupted).

It's more likely the pre-scrub snapshot. It contains you sstable before the 
scrub command. It's unclear if the scrub has done something in your case, but I 
those data are important I would suggest you keep until we figure what is going 
on with your sstables.

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005200#comment-13005200
 ] 

Mike Nadeau commented on CASSANDRA-2307:


Here are my SSTables -
http://184.107.12.190/DATA.tgz

The archive size is 2.5GB, extracted it's around 12GB.

By the way my issue might be a bad hard disk... I saw a couple of those 
exceptions in the log:

org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column 
name length 0
And
Caused by: java.io.IOException: Corrupt (negative) value length encountered

I moved all my data to a hard disk I know to be healty and restarted Cassandra. 
I just hope we can repair everything!

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005209#comment-13005209
 ] 

Sylvain Lebresne commented on CASSANDRA-2307:
-

2.5GB, ouch :)

I'll try to have a look to those sstables. In the meantime, if you have only 
one node having problem and unless your replication factor is 1, you safest and 
probably quicker way to go back to a correct state may well be to replace the 
node with the potentially bad disk.

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at 
 org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
 at

[jira] Commented: (CASSANDRA-2305) Tombstoned rows not purged after gcgraceseconds

[
https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005211#comment-13005211
]

Sylvain Lebresne commented on CASSANDRA-2305:
-

This will also happen if you remove all the columns inside the row (even though
you didn't issued a row deletion command).

The problem is that when you flush + compact and gcGrace has elapsed, if the
row is empty (i.e. all tombstone have been collected), the row itself is
collected. This means that when you issue the second wave of inserts, there is
no trace whatsoever of the row.

That's why you are not supposed to have a gcGrace too low and why it is highly
advised to use the current time as a timestamp. If so, the scenario above will
never happen.

Best thing we can do is probably to edit
http://wiki.apache.org/cassandra/DistributedDeletes to add that gcGrace should
be such that no insert with a timestamp lower that a delete could reach any
given node after gcGrace has elapsed.

Tombstoned rows not purged after gcgraceseconds
---

Key: CASSANDRA-2305
URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Reporter: Jeffrey Wang
Assignee: Sylvain Lebresne
Priority: Minor
Fix For: 0.7.4

From email to list:
I was wondering if this is the expected behavior of deletes (0.7.0). Let's
say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0.
The following sequence of operations happens (in the given order):
insert row X with timestamp T
delete row X with timestamp T+1
force flush + compaction
insert row X with timestamp T
My understanding is that the tombstone created by the delete (and row X) will
disappear with the flush + compaction which means the last insertion should
show up. My experimentation, however, suggests otherwise (the last insertion
does not show up).
I believe I have traced this to the fact that the markedForDeleteAt field on
the ColumnFamily does not get reset after a compaction (after
gc_grace_seconds has passed); is this desirable? I think it introduces an
inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005213#comment-13005213
 ] 

Mike Nadeau commented on CASSANDRA-2307:


My replication factor is 3 and I have 7 other healthy nodes.

Could you help me with the procedure to recover the node? It's the first time I 
face data corruption with Cassandra. One of my concern is that my corrupted 
node is my only node with bootstrap=false, and it's seed of all the other 
nodes. Right now I have only one seed, I never needed more and frankly I'm not 
sure of the rules to calculate how many seeds a cluster should have.

Here's my guess for the procedure, please tell me if I have anything wrong:
1- Stop the corrupted node (let's call it node1)
2- Change bootstrap to false on one another node (let's call it node2), remove 
seed
3- On all other nodes, replace seed (node1) with node2, keep bootstrap to true
4- Restart node2
5- Restart all other nodes, they should be fine
6- Flush node1 data and restart it, it should get its data

I might be worrying too much with the bootstrap/seed settings, I still don't 
understand them at 100%. 

Thanks.

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at

[jira] Commented: (CASSANDRA-2305) Tombstoned rows not purged after gcgraceseconds

[
https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005228#comment-13005228
]

Jonathan Ellis commented on CASSANDRA-2305:
---

bq. The problem is that when you flush + compact and gcGrace has elapsed, if
the row is empty (i.e. all tombstone have been collected), the row itself is
collected. This means that when you issue the second wave of inserts, there is
no trace whatsoever of the row.

That's what's supposed to happen, but Jeffrey is saying that is NOT what he
observes.

Tombstoned rows not purged after gcgraceseconds
---

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1080310 - in /cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra: db/ColumnFamilyStore.java service/StorageService.java

Author: jbellis
Date: Thu Mar 10 18:38:03 2011
New Revision: 1080310

URL: http://svn.apache.org/viewvc?rev=1080310view=rev
Log:
include secondary index CFs when deciding which CFs to flush under memory 
pressure
patch by jbellis; tested by Matt Conway for CASSANDRA-2295

Modified:

cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/ColumnFamilyStore.java

cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/service/StorageService.java

Modified: 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/ColumnFamilyStore.java?rev=1080310r1=1080309r2=1080310view=diff
==
--- 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
 (original)
+++ 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
 Thu Mar 10 18:38:03 2011
@@ -32,6 +32,7 @@ import javax.management.ObjectName;
 
 import com.google.common.collect.Iterables;
 import org.apache.commons.collections.IteratorUtils;
+import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -334,7 +335,6 @@ public class ColumnFamilyStore implement
 {
 public void run()
 {
-logger.info(Creating index {}.{}, table, 
indexedCfMetadata.cfName);
 try
 {
 forceBlockingFlush();
@@ -348,7 +348,6 @@ public class ColumnFamilyStore implement
 throw new AssertionError(e);
 }
 buildSecondaryIndexes(getSSTables(), 
FBUtilities.singleton(info.name));
-logger.info(Index {} complete, indexedCfMetadata.cfName);
 SystemTable.setIndexBuilt(table.name, 
indexedCfMetadata.cfName);
 }
 };
@@ -357,7 +356,8 @@ public class ColumnFamilyStore implement
 
 public void buildSecondaryIndexes(CollectionSSTableReader sstables, 
SortedSetByteBuffer columns)
 {
-logger.debug(Submitting index build to compactionmanager);
+logger.info(String.format(Submitting index build of %s for data in 
%s,
+  metadata.comparator.getString(columns), 
StringUtils.join(sstables, , )));
 Table.IndexBuilder builder = table.createIndexBuilder(this, columns, 
new ReducingKeyIterator(sstables));
 Future future = CompactionManager.instance.submitIndexBuild(this, 
builder);
 try
@@ -374,6 +374,7 @@ public class ColumnFamilyStore implement
 {
 throw new RuntimeException(e);
 }
+logger.info(Index build of  + metadata.comparator.getString(columns) 
+  complete);
 }
 
 // called when dropping or renaming a CF. Performs mbean housekeeping and 
invalidates CFS to other operations.
@@ -684,26 +685,31 @@ public class ColumnFamilyStore implement
 try
 {
 if (oldMemtable.isFrozen())
+{
+logger.debug(memtable is already frozen; another thread must 
be flushing it);
 return null;
+}
 
 boolean isDropped = isIndex()
   ? DatabaseDescriptor.getCFMetaData(table.name, 
getParentColumnfamily()) == null
   : 
DatabaseDescriptor.getCFMetaData(metadata.cfId) == null;
 if (isDropped)
-return null; // column family was dropped. no point in 
flushing.
+{
+logger.debug(column family was dropped; no point in 
flushing);
+return null;
+}
 
 assert memtable == oldMemtable;
 memtable.freeze();
 final CommitLogSegment.CommitLogContext ctx = writeCommitLog ? 
CommitLog.instance.getContext() : null;
-logger.info(switching in a fresh Memtable for  + columnFamily + 
 at  + ctx);
 
 // submit the memtable for any indexed sub-cfses, and our own.
 ListColumnFamilyStore icc = new 
ArrayListColumnFamilyStore(indexedColumns.size());
-icc.add(this);
-for (ColumnFamilyStore indexCfs : indexedColumns.values())
+// don't assume that this.memtable is dirty; forceFlush can bring 
us here during index build even if it is not
+for (ColumnFamilyStore cfs : 
Iterables.concat(Collections.singleton(this), indexedColumns.values()))
 {
-if (!indexCfs.memtable.isClean())
-icc.add(indexCfs);
+if (!cfs.memtable.isClean())
+icc.add(cfs);
 }
 final CountDownLatch latch = new CountDownLatch(icc.size());
 for (ColumnFamilyStore cfs : icc)
@@ -711,6 +717,10 @@ public class ColumnFamilyStore implement

[jira] Commented: (CASSANDRA-2305) Tombstoned rows not purged after gcgraceseconds

[
https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005236#comment-13005236
]

Sylvain Lebresne commented on CASSANDRA-2305:
-

Oups, my mistake. I somehow confused myself. I was not able to reproduce
though, but I'll try harder tomorrow.

Tombstoned rows not purged after gcgraceseconds
---

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2296) Scrub resulting in bloom filter claims to be longer than entire row size error

2011-03-10 Thread Jason Harvey (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005239#comment-13005239
 ] 

Jason Harvey commented on CASSANDRA-2296:
-

Here is the debug output. Going to get a comparison on the unpatched 0.7.3 to 
see if there is any difference.

{code}
DEBUG 11:50:52,510 Reading row at 504216964
DEBUG 11:50:52,510 row 636f6d6d656e74735f706172656e74735f3233383135363235 is 66 
bytes
DEBUG 11:50:52,510 Index doublecheck: row 
636f6d6d656e74735f706172656e74735f3233383135363235 is 66 bytes
 INFO 11:50:52,511 Last written key : 
DecoratedKey(125686934811414729670440675125192621396, 
627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d313237393332313937363529)
 INFO 11:50:52,511 Current key : 
DecoratedKey(11047858886149374835950241979723972473, 
636f6d6d656e74735f706172656e74735f3233383135363235)
 INFO 11:50:52,511 Writing into file 
/var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
 WARN 11:50:52,511 Non-fatal error reading row (stacktrace follows)
java.io.IOException: Keys must be written in ascending order.
at 
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
at 
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
at 
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:598)
at 
org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at 
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
{code}



 Scrub resulting in bloom filter claims to be longer than entire row size 
 error
 

 Key: CASSANDRA-2296
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2296
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Affects Versions: 0.7.3
Reporter: Jason Harvey
Assignee: Jonathan Ellis
 Fix For: 0.7.4

 Attachments: 2296.txt, sstable_part1.tar.bz2, sstable_part2.tar.bz2


 Doing a scrub on a node which I upgraded from 0.7.1 (was previously 0.6.8) to 
 0.7.3. Getting this error multiple times:
 {code}
  WARN [CompactionExecutor:1] 2011-03-08 18:33:52,513 CompactionManager.java 
 (line 625) Row is unreadable; skipping to next
  WARN [CompactionExecutor:1] 2011-03-08 18:33:52,514 CompactionManager.java 
 (line 599) Non-fatal error reading row (stacktrace follows)
 java.io.IOError: java.io.EOFException: bloom filter claims to be longer than 
 entire row size
 at 
 org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:117)
 at 
 org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:590)
 at 
 org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
 at 
 org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 Caused by: java.io.EOFException: bloom filter claims to be longer than entire 
 row size
 at 
 org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:113)
 at 
 org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:87)
 ... 8 more
  WARN [CompactionExecutor:1] 2011-03-08 18:33:52,515 CompactionManager.java 
 (line 625) Row is unreadable; skipping to next
  INFO [CompactionExecutor:1] 2011-03-08 18:33:53,777 CompactionManager.java 
 (line 637) Scrub of 
 SSTableReader(path='/cassandra/data/reddit/Hide-f-671-Data.db') complete: 
 254709 rows in new sstable
  WARN [CompactionExecutor:1] 2011-03-08 18:33:53,777 CompactionManager.java 
 (line 639) Unable to recover 1630 that were skipped.  You can attempt manual 
 recovery from the pre-scrub snapshot.  You can also run nodetool repair to 
 transfer the data from a healthy replica, if any
 {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see:

[jira] Updated: (CASSANDRA-2302) JDBC driver needs ResultSet.getMetaData()

2011-03-10 Thread Gary Dusbabek (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Dusbabek updated CASSANDRA-2302:
-

Attachment: 
v1-0001-RowData-exposes-sparse-column-attributes-through-Cassa.txt

 JDBC driver needs ResultSet.getMetaData()
 -

 Key: CASSANDRA-2302
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2302
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Gary Dusbabek
Assignee: Gary Dusbabek
Priority: Minor
  Labels: cql
 Fix For: 0.8

 Attachments: 
 v1-0001-RowData-exposes-sparse-column-attributes-through-Cassa.txt




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2296) Scrub resulting in bloom filter claims to be longer than entire row size error

2011-03-10 Thread Jason Harvey (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005253#comment-13005253
 ] 

Jason Harvey commented on CASSANDRA-2296:
-

Disregard. Getting the same thing on unpatched 0.7.3. I'll create a separate 
bug report.

 Scrub resulting in bloom filter claims to be longer than entire row size 
 error
 

 Key: CASSANDRA-2296
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2296
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Affects Versions: 0.7.3
Reporter: Jason Harvey
Assignee: Jonathan Ellis
 Fix For: 0.7.4

 Attachments: 2296.txt, sstable_part1.tar.bz2, sstable_part2.tar.bz2


 Doing a scrub on a node which I upgraded from 0.7.1 (was previously 0.6.8) to 
 0.7.3. Getting this error multiple times:
 {code}
  WARN [CompactionExecutor:1] 2011-03-08 18:33:52,513 CompactionManager.java 
 (line 625) Row is unreadable; skipping to next
  WARN [CompactionExecutor:1] 2011-03-08 18:33:52,514 CompactionManager.java 
 (line 599) Non-fatal error reading row (stacktrace follows)
 java.io.IOError: java.io.EOFException: bloom filter claims to be longer than 
 entire row size
 at 
 org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:117)
 at 
 org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:590)
 at 
 org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
 at 
 org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 Caused by: java.io.EOFException: bloom filter claims to be longer than entire 
 row size
 at 
 org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:113)
 at 
 org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:87)
 ... 8 more
  WARN [CompactionExecutor:1] 2011-03-08 18:33:52,515 CompactionManager.java 
 (line 625) Row is unreadable; skipping to next
  INFO [CompactionExecutor:1] 2011-03-08 18:33:53,777 CompactionManager.java 
 (line 637) Scrub of 
 SSTableReader(path='/cassandra/data/reddit/Hide-f-671-Data.db') complete: 
 254709 rows in new sstable
  WARN [CompactionExecutor:1] 2011-03-08 18:33:53,777 CompactionManager.java 
 (line 639) Unable to recover 1630 that were skipped.  You can attempt manual 
 recovery from the pre-scrub snapshot.  You can also run nodetool repair to 
 transfer the data from a healthy replica, if any
 {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (CASSANDRA-2309) Scrub resulting in Keys must be written in ascending order exception

2011-03-10 Thread Jason Harvey (JIRA)

Scrub resulting in Keys must be written in ascending order exception
--

 Key: CASSANDRA-2309
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.3
Reporter: Jason Harvey


Getting the following when I try to scrub. The SSTable causing it is over a 
gig. Trying to repro on smaller tables so I have something to upload.

{code}
DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java 
(line 543) Reading row at 552076476
DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java 
(line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java 
(line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 
is 168 bytes
 INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 
108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 
627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d313237393332313937363529)
 INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 
109) Current key : DecoratedKey(11163129555189411157074346827960371449, 
6c6173745f636f6d6d656e74735f74335f386c387633)
 INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 
110) Writing into file 
/var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
 WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java 
(line 599) Non-fatal error reading row (stacktrace follows)
java.io.IOException: Keys must be written in ascending order.
at 
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
at 
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
at 
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
at 
org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at 
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
{code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (CASSANDRA-2310) CassandraStorage for pig checks for environment variable on mappers/reducers, but it should only need to be set on the machine launching pig.

2011-03-10 Thread Eldon Stegall (JIRA)

CassandraStorage for pig checks for environment variable on mappers/reducers, 
but it should only need to be set on the machine launching pig.
-

 Key: CASSANDRA-2310
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2310
 Project: Cassandra
  Issue Type: Bug
  Components: Contrib
Affects Versions: 0.7.4, 0.8
Reporter: Eldon Stegall


Only error out if necessary pig settings have not previously been set in job 
config. CassandraStorage checks for environment variables on mappers/reducers, 
but it should only need to be set on the machine launching the pig jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2310) CassandraStorage for pig checks for environment variable on mappers/reducers, but it should only need to be set on the machine launching pig.

2011-03-10 Thread Eldon Stegall (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eldon Stegall updated CASSANDRA-2310:
-

Attachment: 0001-Dont-fail-if-configs-already-set.patch

 CassandraStorage for pig checks for environment variable on mappers/reducers, 
 but it should only need to be set on the machine launching pig.
 -

 Key: CASSANDRA-2310
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2310
 Project: Cassandra
  Issue Type: Bug
  Components: Contrib
Affects Versions: 0.7.4, 0.8
Reporter: Eldon Stegall
 Attachments: 0001-Dont-fail-if-configs-already-set.patch


 Only error out if necessary pig settings have not previously been set in job 
 config. CassandraStorage checks for environment variables on 
 mappers/reducers, but it should only need to be set on the machine launching 
 the pig jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2309) Scrub resulting in Keys must be written in ascending order exception


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005264#comment-13005264
 ] 

Jonathan Ellis commented on CASSANDRA-2309:
---

Well, it's right -- the current key sorts before the last written key so 
it's out of order.

If scrub is able to continue, then you should probably do an sstable2json, then 
let scrub go ahead and strip out those out-of-order keys, then use the log 
messages to pull the keys that were out of order out of your json dump.

 Scrub resulting in Keys must be written in ascending order exception
 --

 Key: CASSANDRA-2309
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.3
Reporter: Jason Harvey

 Getting the following when I try to scrub. The SSTable causing it is over a 
 gig. Trying to repro on smaller tables so I have something to upload.
 {code}
 DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java 
 (line 543) Reading row at 552076476
 DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java 
 (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
 DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java 
 (line 581) Index doublecheck: row 
 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
  INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 
 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 
 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d313237393332313937363529)
  INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 
 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 
 6c6173745f636f6d6d656e74735f74335f386c387633)
  INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 
 110) Writing into file 
 /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
  WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java 
 (line 599) Non-fatal error reading row (stacktrace follows)
 java.io.IOException: Keys must be written in ascending order.
 at 
 org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
 at 
 org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
 at 
 org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
 at 
 org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:636)
 {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2310) CassandraStorage for pig checks for environment variable on mappers/reducers, but it should only need to be set on the machine launching pig.


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2310:
--

 Reviewer: brandon.williams
  Component/s: Hadoop
 Priority: Minor  (was: Major)
Affects Version/s: (was: 0.8)
Fix Version/s: 0.7.4

 CassandraStorage for pig checks for environment variable on mappers/reducers, 
 but it should only need to be set on the machine launching pig.
 -

 Key: CASSANDRA-2310
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2310
 Project: Cassandra
  Issue Type: Bug
  Components: Contrib, Hadoop
Affects Versions: 0.7.4
Reporter: Eldon Stegall
Priority: Minor
 Fix For: 0.7.4

 Attachments: 0001-Dont-fail-if-configs-already-set.patch


 Only error out if necessary pig settings have not previously been set in job 
 config. CassandraStorage checks for environment variables on 
 mappers/reducers, but it should only need to be set on the machine launching 
 the pig jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (CASSANDRA-2311) type validated row keys

2011-03-10 Thread Eric Evans (JIRA)

type validated row keys
---

 Key: CASSANDRA-2311
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2311
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Eric Evans
Assignee: Jon Hermes
 Fix For: 0.8


The idea here is to allow the assignment of a column-family-wide key type used 
to perform validation, (similar to how default_validation_class does for column 
values).

This should be as straightforward as extending the column family schema to 
include the new attribute, and updating {{ThriftValidation.validateKey}} to 
validate the key ({{AbstractType.validate}}).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2310) CassandraStorage for pig checks for environment variable on mappers/reducers, but it should only need to be set on the machine launching pig.

2011-03-10 Thread Jeremy Hanna (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005275#comment-13005275
 ] 

Jeremy Hanna commented on CASSANDRA-2310:
-

+1 - works where before it would error out in mapreduce mode (post storefunc). 
I did suggest maybe unifying the conditions and using an else or something to 
make it more concise, but that's a trivial thing.

 CassandraStorage for pig checks for environment variable on mappers/reducers, 
 but it should only need to be set on the machine launching pig.
 -

 Key: CASSANDRA-2310
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2310
 Project: Cassandra
  Issue Type: Bug
  Components: Contrib, Hadoop
Affects Versions: 0.7.4
Reporter: Eldon Stegall
Priority: Minor
 Fix For: 0.7.4

 Attachments: 0001-Dont-fail-if-configs-already-set.patch


 Only error out if necessary pig settings have not previously been set in job 
 config. CassandraStorage checks for environment variables on 
 mappers/reducers, but it should only need to be set on the machine launching 
 the pig jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2302) JDBC driver needs ResultSet.getMetaData()

2011-03-10 Thread Gary Dusbabek (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Dusbabek updated CASSANDRA-2302:
-

Attachment: 
v1-0001-RowData-exposes-sparse-column-attributes-through-Cassa.txt

 JDBC driver needs ResultSet.getMetaData()
 -

 Key: CASSANDRA-2302
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2302
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Gary Dusbabek
Assignee: Gary Dusbabek
Priority: Minor
  Labels: cql
 Fix For: 0.8

 Attachments: 
 v1-0001-RowData-exposes-sparse-column-attributes-through-Cassa.txt




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction

2011-03-10 Thread JIRA


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005295#comment-13005295
 ] 

Sébastien Giroux commented on CASSANDRA-2307:
-

org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column 
name length 0

This is the exception I had BEFORE running scrub. Scrub is supposed to rebuild 
your sstable while fixing some corruption issues (if any). Are you sure scrub 
is done running on all problematic sstable when you see this exception ? It can 
takes a little while to run, especially on large dataset.

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005298#comment-13005298
 ] 

Mike Nadeau commented on CASSANDRA-2307:


Yes the scrub finished running, and it seems to have done a lot of stuff.. but 
I still get the exception.

I'll go ahead and try to replace the node.

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at 
 org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
 at 
 org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
 at

[jira] Commented: (CASSANDRA-1938) Use UUID as node identifiers in counters instead of IP addresses


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005310#comment-13005310
 ] 

T Jake Luciani commented on CASSANDRA-1938:
---

Ok cluster testing complete and all looks good!

I just noticed a SerializationTest started failing though, Can you take a look

{code}
[junit] Testcase: 
testRowMutationRead(org.apache.cassandra.db.SerializationsTest):  FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at 
org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:233)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:191)
[junit] at 
org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:224)
[junit] at 
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:506)
[junit] at 
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:487)
[junit] at org.apache.cassandra.db.Table.initCf(Table.java:358)
[junit] at org.apache.cassandra.db.Table.init(Table.java:295)
[junit] at org.apache.cassandra.db.Table.open(Table.java:112)
[junit] at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:132)
[junit] at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:125)
[junit] at 
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:365)
[junit] at 
org.apache.cassandra.db.SerializationsTest.testRowMutationRead(SerializationsTest.java:219)

{code}

 Use UUID as node identifiers in counters instead of IP addresses 
 -

 Key: CASSANDRA-1938
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1938
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
 Fix For: 0.8

 Attachments: 0001-Use-uuid-instead-of-IP-for-counters.patch, 
 0002-Merge-old-shard-locally.patch, 0003-Thrift-change-to-CfDef.patch, 
 1938-rebase-and-fix-v1.txt, 1938_discussion

   Original Estimate: 56h
  Remaining Estimate: 56h

 The use of IP addresses as node identifiers in the partition of a given
 counter is fragile. Changes of the node's IP addresses can result in data
 loss. This patch proposes to use UUIDs instead.
 NOTE: this breaks the on-disk file format (for counters)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1080338 - in /cassandra/trunk: ./ conf/ contrib/ contrib/pig/ contrib/pig/bin/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/db/ src/java/org/apache

Author: jbellis
Date: Thu Mar 10 20:28:17 2011
New Revision: 1080338

URL: http://svn.apache.org/viewvc?rev=1080338view=rev
Log:
merge from 0.7

Modified:
cassandra/trunk/   (props changed)
cassandra/trunk/CHANGES.txt
cassandra/trunk/build.xml
cassandra/trunk/conf/cassandra.yaml
cassandra/trunk/contrib/   (props changed)
cassandra/trunk/contrib/pig/bin/pig_cassandra
cassandra/trunk/contrib/pig/build.xml

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java
   (props changed)
cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLog.java
cassandra/trunk/src/java/org/apache/cassandra/service/StorageService.java
cassandra/trunk/src/java/org/apache/cassandra/tools/SSTableExport.java
cassandra/trunk/test/unit/org/apache/cassandra/db/CommitLogTest.java

Propchange: cassandra/trunk/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Thu Mar 10 20:28:17 2011
@@ -1,5 +1,5 @@
 
/cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1071777,1076891
-/cassandra/branches/cassandra-0.7:1026516-1079936
+/cassandra/branches/cassandra-0.7:1026516-1080312
 /cassandra/branches/cassandra-0.7.0:1053690-1055654
 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689
 /incubator/cassandra/branches/cassandra-0.3:774578-796573

Modified: cassandra/trunk/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1080338r1=1080337r2=1080338view=diff
==
--- cassandra/trunk/CHANGES.txt (original)
+++ cassandra/trunk/CHANGES.txt Thu Mar 10 20:28:17 2011
@@ -27,6 +27,9 @@
  * avoid writing empty rows when scrubbing tombstoned rows (CASSANDRA-2296)
  * fix assertion error in range and index scans for CL  ALL
(CASSANDRA-2282)
+ * fix commitlog replay when flush position refers to data that didn't
+   get synced before server died (CASSANDRA-2285)
+ * fix fd leak in sstable2json with non-mmap'd i/o (CASSANDRA-2304)
 
 
 0.7.3

Modified: cassandra/trunk/build.xml
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/build.xml?rev=1080338r1=1080337r2=1080338view=diff
==
--- cassandra/trunk/build.xml (original)
+++ cassandra/trunk/build.xml Thu Mar 10 20:28:17 2011
@@ -631,7 +631,16 @@
   fileset dir=@{inputdir} includes=@{filter} /
 /batchtest
   /junit
-  fail if=testfailed message=Some @{suitename} test(s) failed./
+  fail message=Some @{suitename} test(s) failed.
+condition
+and
+isset property=testfailed/
+not
+  isset property=ant.test.failure.ignore/
+/not
+  /and
+/condition
+  /fail
 /sequential
   /macrodef
 

Modified: cassandra/trunk/conf/cassandra.yaml
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/conf/cassandra.yaml?rev=1080338r1=1080337r2=1080338view=diff
==
--- cassandra/trunk/conf/cassandra.yaml (original)
+++ cassandra/trunk/conf/cassandra.yaml Thu Mar 10 20:28:17 2011
@@ -87,7 +87,7 @@ commitlog_rotation_threshold_in_mb: 128
 # performing the sync.
 commitlog_sync: periodic
 
-# the other option is timed, where writes may be acked immediately
+# the other option is periodic where writes may be acked immediately
 # and the CommitLog is simply synced every commitlog_sync_period_in_ms
 # milliseconds.
 commitlog_sync_period_in_ms: 1

Propchange: cassandra/trunk/contrib/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Thu Mar 10 20:28:17 2011
@@ -1,5 +1,5 @@
 
/cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009
-/cassandra/branches/cassandra-0.7/contrib:1026516-1079936
+/cassandra/branches/cassandra-0.7/contrib:1026516-1080312
 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654
 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689
 /incubator/cassandra/branches/cassandra-0.3/contrib:774578-796573

Modified: cassandra/trunk/contrib/pig/bin/pig_cassandra
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/contrib/pig/bin/pig_cassandra?rev=1080338r1=1080337r2=1080338view=diff

[jira] Commented: (CASSANDRA-1938) Use UUID as node identifiers in counters instead of IP addresses


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005315#comment-13005315
 ] 

Jonathan Ellis commented on CASSANDRA-1938:
---

that's been failing for a while, it's not this patch causing it.

 Use UUID as node identifiers in counters instead of IP addresses 
 -

 Key: CASSANDRA-1938
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1938
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
 Fix For: 0.8

 Attachments: 0001-Use-uuid-instead-of-IP-for-counters.patch, 
 0002-Merge-old-shard-locally.patch, 0003-Thrift-change-to-CfDef.patch, 
 1938-rebase-and-fix-v1.txt, 1938_discussion

   Original Estimate: 56h
  Remaining Estimate: 56h

 The use of IP addresses as node identifiers in the partition of a given
 counter is fragile. Changes of the node's IP addresses can result in data
 loss. This patch proposes to use UUIDs instead.
 NOTE: this breaks the on-disk file format (for counters)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2295) memory pressure flusher should include secondary index CFs


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005314#comment-13005314
 ] 

Hudson commented on CASSANDRA-2295:
---

Integrated in Cassandra-0.7 #371 (See 
[https://hudson.apache.org/hudson/job/Cassandra-0.7/371/])
include secondary index CFs when deciding which CFs to flush under memory 
pressure
patch by jbellis; tested by Matt Conway for CASSANDRA-2295


 memory pressure flusher should include secondary index CFs
 --

 Key: CASSANDRA-2295
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2295
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 0.7.1
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.7.4

 Attachments: 2295-v2.txt, 2295-v3.txt, 2295-v4.txt, 2295.txt




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2299) Add counters to stress tool


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005318#comment-13005318
 ] 

Jonathan Ellis commented on CASSANDRA-2299:
---

committed by tjake

 Add counters to stress tool
 ---

 Key: CASSANDRA-2299
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2299
 Project: Cassandra
  Issue Type: Improvement
  Components: Contrib
Affects Versions: 0.8
Reporter: T Jake Luciani
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8

 Attachments: 0001-Add-counter-support-to-stress-java-v2.patch, 
 0001-Add-counter-support-to-stress-java.patch, 
 0001-CASSANDRA-2299-Alternate-counter-stress-impl.txt, help-message-fix.patch

  Time Spent: 4h
  Remaining Estimate: 0h

 For purposes of profiling changes it would be very helpful to have counter 
 operations added to the stress tool.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005316#comment-13005316
 ] 

Jonathan Ellis commented on CASSANDRA-2307:
---

are you using TTLs in this CF?

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at 
 org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
 at 
 org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
 at 
 org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
 at 
 org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217)
 at

[jira] Commented: (CASSANDRA-1938) Use UUID as node identifiers in counters instead of IP addresses


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005319#comment-13005319
 ] 

T Jake Luciani commented on CASSANDRA-1938:
---

+1, nice work

 Use UUID as node identifiers in counters instead of IP addresses 
 -

 Key: CASSANDRA-1938
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1938
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
 Fix For: 0.8

 Attachments: 0001-Use-uuid-instead-of-IP-for-counters.patch, 
 0002-Merge-old-shard-locally.patch, 0003-Thrift-change-to-CfDef.patch, 
 1938-rebase-and-fix-v1.txt, 1938_discussion

   Original Estimate: 56h
  Remaining Estimate: 56h

 The use of IP addresses as node identifiers in the partition of a given
 counter is fragile. Changes of the node's IP addresses can result in data
 loss. This patch proposes to use UUIDs instead.
 NOTE: this breaks the on-disk file format (for counters)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005331#comment-13005331
 ] 

Mike Nadeau commented on CASSANDRA-2307:


No TTL.

It's a very simple CF for which we generate a unique primary key (no timeUUID). 
And each document is very small, maybe around 300 bytes total in average.

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at 
 org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
 at 
 org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
 at

[jira] Commented: (CASSANDRA-2307) IndexOutOfBoundsException during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005339#comment-13005339
 ] 

Mike Nadeau commented on CASSANDRA-2307:


No deletes.

 IndexOutOfBoundsException during compaction
 ---

 Key: CASSANDRA-2307
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2, 0.7.3
 Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned 
 to Cassandra, JRE 1.6.0_24
Reporter: Mike Nadeau

 We're getting an IndexOutOfBounds exception when compacting.
 Here's the detailed error we get on-screen when running nodetool -h 
 10.3.133.10 compact:
 Error occured while compacting keyspace test
 java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at 
 org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
 Source)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
 Source)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
 Source)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: java.lang.IndexOutOfBoundsException
 at java.nio.Buffer.checkIndex(Unknown Source)
 at java.nio.HeapByteBuffer.getInt(Unknown Source)
 at 
 org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
 at org.apache.cassandra.io.PrecompactedRow.init(PrecompactedRow.java:94)
 at 
 org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
 at 
 org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
 at 
 org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at 
 org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
 at 
 org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
 at 
 org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
 at 
 org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217)
 at

[jira] Commented: (CASSANDRA-2221) 'show create' commands on the CLI to export schema

2011-03-10 Thread Jon Hermes (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005343#comment-13005343
]

Jon Hermes commented on CASSANDRA-2221:
---

CASSANDRA-1906 completes the YAML removal (all that's left is with on-boot yaml
load of per-node settings, which is fine as is).

Tying this right next to the 'show keyspaces;' logic (maybe 'export
keyspaces;') is going to be the easiest, as it already iterates through your
settings.

Instead of showing shorthand ( default_validation_class: BytesType), show
longhand (with default_validation_class=BytesType ) and cascade them into a
create column family ... output.

'show create' commands on the CLI to export schema
--

Key: CASSANDRA-2221
URL: https://issues.apache.org/jira/browse/CASSANDRA-2221
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Jeremy Hanna
Assignee: Aaron Morton
Priority: Minor
Labels: cli
Fix For: 0.7.4

It would be nice to have 'show create' type of commands on the command-line
so that it would generate the DDL for the schema.
A scenario that would make this useful is where a team works out a data model
over time with a dev cluster. They want to use parts of that schema for new
clusters that they create, like a staging/prod cluster. It would be very
handy in this scenario to have some sort of export mechanism.
Another use case is for testing purposes - you want to replicate a problem.
We currently have schematool for import/export but that is deprecated and it
exports into yaml.
This new feature would just be able to 'show' - or export if they want the
entire keyspace - into a script or commands that could be used in a cli
script. It would need to be able to regenerate everything about the keyspace
including indexes and metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2312) Stress.java columns are bigger than advertised


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2312:
--

Attachment: 2312.txt

Patch to leave column values as raw bytes. Also adds comparator metadata for 
column names.

 Stress.java columns are bigger than advertised 
 ---

 Key: CASSANDRA-2312
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2312
 Project: Cassandra
  Issue Type: Bug
  Components: Contrib, Tools
Affects Versions: 0.7.0
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.7.4

 Attachments: 2312.txt


 Converting from bytes to hex makes the columns 4x larger than they should be. 
  (2x for conversion to hex, then another 2x for converting to UTF-16 which is 
 the default String encoding.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2294) secondary index CFs should use parent CF flush thresholds


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2294:
--

Attachment: 2294-v2.txt

New patch applies flush thresholds changed via update column family to 
indexes as well (rather than waiting until next restart).

I created 1M rows with stress.java, ran update column family standard1 with 
memtable_operations=0.0001, created an index, and the index started building 
with flushes every 100 rows.  I changed it to memtable_operations=0.01 and it 
started flushing every 1 rows.

 secondary index CFs should use parent CF flush thresholds
 -

 Key: CASSANDRA-2294
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2294
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 0.7.0
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.7.4

 Attachments: 2294-v2.txt, 2294.txt




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2301) OOM on repair with many inconsistent ranges

2011-03-10 Thread Joaquin Casares (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005414#comment-13005414
 ] 

Joaquin Casares commented on CASSANDRA-2301:


+1 on v3.

Ran the patched code twice and my cluster of 2 didn't OOM.

 OOM on repair with many inconsistent ranges
 ---

 Key: CASSANDRA-2301
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2301
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Joaquin Casares
Assignee: Joaquin Casares
 Fix For: 0.7.4

 Attachments: 2301-v2.txt, 2301-v3.txt, 2301.diff


 Repair can OOM when lots of ranges are inconsistent, causing many sstables to 
 be streamed.
 I replicated the error by making 1264 3MB sstables on one node, added a 
 second node, changed the replication factor to 2, and ran a repair.
 Looking at the heap dump of the original failure, there were 2.4GB of 
 FutureTasks, each taking up 8MB of space. I tracked down the 
 BufferedRandomAccessFile and made sure that it was cleared every time it was 
 closed inside of src/java/org/apache/cassandra/io/sstable/SSTableWriter.java.
 Attached is the patch I used which stopped the error when I was trying to 
 replicate it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Cassandra Wiki] Update of LargeDataSetConsiderations_JP by MakiWatanabe

2011-03-10 Thread Apache Wiki

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The LargeDataSetConsiderations_JP page has been changed by MakiWatanabe.
The comment on this change is: Translation to Japanese completed.
http://wiki.apache.org/cassandra/LargeDataSetConsiderations_JP?action=diffrev1=22rev2=23

--

  
* 
将来的な改善方法については以下のリンクで議論されています:[[https://issues.apache.org/jira/browse/CASSANDRA-1876|CASSANDRA-1876]],
 [[https://issues.apache.org/jira/browse/CASSANDRA-1881|CASSANDRA-1881]]
  
-  * 
ファイルシステムの選択について:巨大なファイルの削除は、例えばext2/ext3では恐ろしく遅く、多量のseekを要します。xfsまたはext4fsの使用を検討してください。これは恒常的に生じるsstableのバックグラウンドでのunlinkに影響します。また起動速度にも影響します。
+  * 
ファイルシステムの選択について:巨大なファイルの削除は、例えばext2/ext3では恐ろしく遅く、多量のseekを要します。xfsまたはext4fsの使用を検討してください。これは恒常的に生じるsstableのバックグラウンドでのunlinkに影響します。また起動速度にも影響します。(起動時に削除待機中のsstableがある場合、起動プロセスの一環としてそれらを削除します。これは数TBのsstableを削除するような場合は有害でしょう。)
- 
(起動時に削除待機中のsstableがある場合、起動プロセスの一環としてそれらを削除します。従って数TBのsstableを削除するような場合は有害になるでしょう。)
  
-  * 各ノードが多量のデータを格納している場合、ノードの追加には時間がかかります。
+  * 各ノードが多量のデータを格納している場合、ノードの追加には時間がかかります。システムがぎりぎりまで逼迫する前にノードを増設した方がいいでしょう。
  
- Plan for this; do not try to throw additional hardware at a cluster at the 
last minute.
- システムがぎりぎりまで逼迫してからノードを追加するのは避けたほうがいいでしょう。
+  * Cassandraは起動時にstable 
indexファイルを読みます。これは「indexサンプリング」と呼ばれています。indexサンプリングによりキーのサブセット（デフォルトでは100分の1）とディスク上の位置がメモリ上のインデックスに保持されます（[[ArchitectureInternals]]参照）。これはインデックスファイルが大きくなるにつれ、サンプリングに要する時間が長くなることを意味します。従って、キーを多量に含む巨大なインデックスが存在する場合、起動時のindexサンプリングが問題になる可能性があります。
+  * 
行キャッシュを大きくした場合のデメリットは起動に要する時間です。行キャッシュの情報を定期的に保存する際には、キャッシュされているキー値のみが保存されます。キーに対応するデータは起動時にプリフェッチされていなければなりません。巨大なデータセットではこれには多量のseekを要し、行キャッシュが使用可能になるまでの時間は行キャッシュサイズに比例します。（seek
 IOがディスク最適化の影響を受けておらず、かつ十分に大きなデータセットを扱う場合）
+   * 将来的な改善方法については以下のリンクで議論されています: 
[[https://issues.apache.org/jira/browse/CASSANDRA-1625|CASSANDRA-1625]]
  
-  * Cassandra will read through sstable index files on start-up, doing what is 
known as index sampling. This is used to keep a subset (currently and by 
default, 1 out of 100) of keys and and their on-disk location in the index, in 
memory. See [[ArchitectureInternals]]. This means that the larger the index 
files are, the longer it takes to perform this sampling. Thus, for very large 
indexes (typically when you have a very large number of keys) the index 
sampling on start-up may be a significant issue.
-  * A negative side-effect of a large row-cache is start-up time. The periodic 
saving of the row cache information only saves the keys that are cached; the 
data has to be pre-fetched on start-up. On a large data set, this is probably 
going to be seek-bound and the time it takes to warm up the row cache will be 
linear with respect to the row cache size (assuming sufficiently large amounts 
of data that the seek bound I/O is not subject to optimization by disks).
-   * Potential future improvement: 
[[https://issues.apache.org/jira/browse/CASSANDRA-1625|CASSANDRA-1625]].
-

[Cassandra Wiki] Trivial Update of FrontPage_JP by MakiWatanabe

2011-03-10 Thread Apache Wiki

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The FrontPage_JP page has been changed by MakiWatanabe.
http://wiki.apache.org/cassandra/FrontPage_JP?action=diffrev1=73rev2=74

--

   * [[MemtableThresholds|Memtableの閾値]] と 
[[PerformanceTuning_JP|パフォーマンスチューニング]]（翻訳済み）
   * [[CassandraHardware_JP|Cassandraで使うハードウェアの注意点]]（翻訳済み）
   * [[CloudConfig|RackspaceまたはAmazon Web Servicesの構成]]
-  * [[LargeDataSetConsiderations_JP|巨大なデータセットに関する注意点]]（翻訳済み）
+  * [[LargeDataSetConsiderations_JP|多量のデータを扱う場合の注意点]]（翻訳済み）
  
  == クライアントライブラリーに関する開発者向け情報 ==
   * [[API|Thrift API Documentation]] (In progress)

[jira] Commented: (CASSANDRA-2214) Bootstrap Token collision after nodetool loadbalance

2011-03-10 Thread Jaakko Laine (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005463#comment-13005463
 ] 

Jaakko Laine commented on CASSANDRA-2214:
-

Do you have 10.0.0.2 logs from loadbalance period (and especially the parts 
when it chose the new token)?


 Bootstrap Token collision after nodetool loadbalance
 

 Key: CASSANDRA-2214
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2214
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.2
Reporter: Matthias Keller
Assignee: Nick Bailey

 I had two nodes for testing. They both owned around 50% of all data, my test 
 CF has RF=2
 Then I added a third node (bootstrapped it):
 {noformat}
 Address Status State   LoadOwnsToken

 101483442157567999664061592210059906302
 10.0.0.2Up Normal  320.42 KB   50.00%  
 1641285042783798217940352117853438
 10.0.0.3Up Normal  341.53 KB   26.25%  
 61078635599166706937511052402724559481
 10.0.0.1Up Normal  321.3 KB23.75%  
 101483442157567999664061592210059906302
 {noformat}
 Then I wanted to re-balance node 2 (10.0.0.2), so I issued the  loadbalance  
 command on it. It took quite a while but after leaving the ring and coming 
 back, it seems to have assigned the same token as node 3 (10.0.0.3); I get 
 this on node3:
 {quote}
 ERROR 11:47:09,719 Fatal exception in thread Thread[HintedHandoff:1,1,main]
 java.lang.RuntimeException: java.lang.NullPointerException
 at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.*NullPointerException*
 at 
 org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:250)
 at 
 org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:267)
 at 
 org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
 at 
 org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:391)
 at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
 ... 3 more
 {quote}
 And a second later, this on node 1 (10.0.0.1):
 {quote}
 ERROR 11:47:10,719 Fatal exception in thread Thread[GossipStage:2,5,main]
 java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and 
 /10.0.0.2 (token 61078635599166706937511052402724559481
 at 
 org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
 at 
 org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:696)
 at 
 org.apache.cassandra.service.StorageService.onChange(StorageService.java:638)
 at 
 org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1114)
 at 
 org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:639)
 at org.apache.cassandra.gms.Gossiper.handleNewJoin(Gossiper.java:614)
 at 
 org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:686)
 at 
 org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
 at 
 org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1080433 - in /cassandra/branches/cassandra-0.7: CHANGES.txt src/java/org/apache/cassandra/db/CompactionManager.java src/java/org/apache/cassandra/io/sstable/SSTableWriter.java

Author: jbellis
Date: Fri Mar 11 02:57:15 2011
New Revision: 1080433

URL: http://svn.apache.org/viewvc?rev=1080433view=rev
Log:
reduce memory use during streaming of multiple sstables
patch by jbellis; reviewed by mdennis and tested by Joaquin Casares for 
CASSANDRA-2301

Modified:
cassandra/branches/cassandra-0.7/CHANGES.txt

cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java

cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java

Modified: cassandra/branches/cassandra-0.7/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/CHANGES.txt?rev=1080433r1=1080432r2=1080433view=diff
==
--- cassandra/branches/cassandra-0.7/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.7/CHANGES.txt Fri Mar 11 02:57:15 2011
@@ -19,6 +19,7 @@
  * fix commitlog replay when flush position refers to data that didn't
get synced before server died (CASSANDRA-2285)
  * fix fd leak in sstable2json with non-mmap'd i/o (CASSANDRA-2304)
+ * reduce memory use during streaming of multiple sstables (CASSANDRA-2301)
 
 
 0.7.3

Modified: 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java?rev=1080433r1=1080432r2=1080433view=diff
==
--- 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
 (original)
+++ 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
 Fri Mar 11 02:57:15 2011
@@ -927,7 +927,7 @@ public class CompactionManager implement
 return executor.submit(runnable);
 }
 
-public FutureSSTableReader submitSSTableBuild(Descriptor desc)
+public FutureSSTableReader submitSSTableBuild(final Descriptor desc)
 {
 // invalid descriptions due to missing or dropped CFS are handled by 
SSTW and StreamInSession.
 final SSTableWriter.Builder builder = 
SSTableWriter.createBuilder(desc);
@@ -938,7 +938,7 @@ public class CompactionManager implement
 compactionLock.lock();
 try
 {
-executor.beginCompaction(builder.cfs.columnFamily, 
builder);
+executor.beginCompaction(desc.cfname, builder);
 return builder.build();
 }
 finally

Modified: 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java?rev=1080433r1=1080432r2=1080433view=diff
==
--- 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
 (original)
+++ 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
 Fri Mar 11 02:57:15 2011
@@ -244,7 +244,7 @@ public class SSTableWriter extends SSTab
 public static class Builder implements ICompactionInfo
 {
 private final Descriptor desc;
-public final ColumnFamilyStore cfs;
+private final ColumnFamilyStore cfs;
 private BufferedRandomAccessFile dfile;
 
 public Builder(Descriptor desc)
@@ -252,6 +252,14 @@ public class SSTableWriter extends SSTab
 
 this.desc = desc;
 cfs = Table.open(desc.ksname).getColumnFamilyStore(desc.cfname);
+}
+
+// lazy-initialize the file to avoid opening it until it's actually 
executing on the CompactionManager,
+// since the 8MB buffers can use up heap quickly
+private void maybeOpenFile()
+{
+if (dfile != null)
+return;
 try
 {
 dfile = new BufferedRandomAccessFile(new 
File(desc.filenameFor(SSTable.COMPONENT_DATA)), r, 8 * 1024 * 1024, true);
@@ -266,6 +274,8 @@ public class SSTableWriter extends SSTab
 {
 if (cfs.isInvalid())
 return null;
+maybeOpenFile();
+
 File ifile = new File(desc.filenameFor(SSTable.COMPONENT_INDEX));
 File ffile = new File(desc.filenameFor(SSTable.COMPONENT_FILTER));
 assert !ifile.exists();
@@ -332,8 +342,10 @@ public class SSTableWriter extends SSTab
 
 public long getTotalBytes()
 {
+maybeOpenFile();
 try
 {
+// (length is still valid post-close)
 return dfile.length();
 }
 catch (IOException e)
@@ -344,6 +356,8 @@ public class SSTableWriter extends SSTab
 
 public long getBytesComplete()
 {
+

[jira] Resolved: (CASSANDRA-2301) OOM on repair with many inconsistent ranges


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis resolved CASSANDRA-2301.
---

Resolution: Fixed

committed

 OOM on repair with many inconsistent ranges
 ---

 Key: CASSANDRA-2301
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2301
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Joaquin Casares
Assignee: Joaquin Casares
 Fix For: 0.7.4

 Attachments: 2301-v2.txt, 2301-v3.txt, 2301.diff


 Repair can OOM when lots of ranges are inconsistent, causing many sstables to 
 be streamed.
 I replicated the error by making 1264 3MB sstables on one node, added a 
 second node, changed the replication factor to 2, and ran a repair.
 Looking at the heap dump of the original failure, there were 2.4GB of 
 FutureTasks, each taking up 8MB of space. I tracked down the 
 BufferedRandomAccessFile and made sure that it was cleared every time it was 
 closed inside of src/java/org/apache/cassandra/io/sstable/SSTableWriter.java.
 Attached is the patch I used which stopped the error when I was trying to 
 replicate it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2013) Add CL.TWO, CL.THREE; tweak CL documentation

[
https://issues.apache.org/jira/browse/CASSANDRA-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis updated CASSANDRA-2013:
--

Attachment: 2013-assert.txt

Looks like the easiest fix is to just remove the assert, which is somewhat
obsolete anyway. Then assureSufficientLiveNodes will throw
UnavailableException, if necessary.

Add CL.TWO, CL.THREE; tweak CL documentation

Key: CASSANDRA-2013
URL: https://issues.apache.org/jira/browse/CASSANDRA-2013
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Peter Schuller
Assignee: Peter Schuller
Priority: Minor
Fix For: 0.7.4

Attachments: 2013-assert.txt, 2013.txt

Attaching draft patch to add CL.TWO and CL.THREE.
Motivation for adding is that having to select between either ONE or QUORUM
is too narrow a choice for clusters with RF 3. In such a case, it makes
particular sense to want to do writes at e.g. CL.TWO for durability purposes
even though you are not looking to get strong consistency with QUORUM.
CL.THREE is the same argument. TWO and THREE felt reasonable; there is no
objective reason why stopping at THREE is the obvious choice.
Technically one would want to specify an arbitrary number, but that is a much
more significant change.
Two open questions:
(1) I adjusted the documentation of ConsistencyLevel to be more consistent
and also to reflect what I believe to be reality (for example, as far as I
can tell QUORUM doesn't send requests to all nodes as claimed in the .thrift
file). I'm not terribly confident that I have not missed something though.
(2) There is at least one unresolved issue, which is this assertion check
WriteResponseHandler:
assert 1 = blockFor blockFor = 2 *
Table.open(table).getReplicationStrategy().getReplicationFactor()
: String.format(invalid response count %d for replication factor
%d,
blockFor,
Table.open(table).getReplicationStrategy().getReplicationFactor());
At THREE, this causes an assertion failure on keyspace with RF=1. I would, as
a user, expect UnavailableException. However I am uncertain as to what to do
about this assertion. I think this highlights one TWO/THREE are different
from previously existing CL:s, in that they essentially hard-code replicate
counts rather than expressing them in terms that can by definition be served
by the cluster at any RF.
Given that with THREE (and not TWO, but that is only due to the
implementation detail that bootstrapping is involved) implies a replicate
count that is independent of the replication factor, there is essentially a
new failure mode. It is suddenly possible for a consistency level to be
fundamentally incompatible with the RF. My gut reaction is to want
UnavailableException still, and that the assertion check can essentially be
removed (other than the = 1 part).
If a different failure mode is desired, presumably it would not be an
assertion failure (which should indicate a Cassandra bug). Maybe
UnstisfiableConsistencyLevel? I propose just adjusting the assertion (which
has no equivalent in ReadCallback btw); giving a friendlier error message in
case of a CL/RF mismatch would be good, but doesn't feel worth introducing
extra complexity to deal with it.
'ant test' passes. I have tested w/ py_stress with a three-node cluster and
an RF=3 keyspace and with 1 and 2 nodes down, and get expected behavior
(available or unavailable as a function of nodes that are up).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2013) Add CL.TWO, CL.THREE; tweak CL documentation

[
https://issues.apache.org/jira/browse/CASSANDRA-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005470#comment-13005470
]

Jonathan Ellis commented on CASSANDRA-2013:
---

(patch attached to remove assert)

Add CL.TWO, CL.THREE; tweak CL documentation

Attachments: 2013-assert.txt, 2013.txt

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2310) CassandraStorage for pig checks for environment variable on mappers/reducers, but it should only need to be set on the machine launching pig.


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005474#comment-13005474
 ] 

Hudson commented on CASSANDRA-2310:
---

Integrated in Cassandra-0.7 #373 (See 
[https://hudson.apache.org/hudson/job/Cassandra-0.7/373/])


 CassandraStorage for pig checks for environment variable on mappers/reducers, 
 but it should only need to be set on the machine launching pig.
 -

 Key: CASSANDRA-2310
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2310
 Project: Cassandra
  Issue Type: Bug
  Components: Contrib, Hadoop
Affects Versions: 0.7.4
Reporter: Eldon Stegall
Priority: Minor
 Fix For: 0.7.4

 Attachments: 0001-Dont-fail-if-configs-already-set.patch


 Only error out if necessary pig settings have not previously been set in job 
 config. CassandraStorage checks for environment variables on 
 mappers/reducers, but it should only need to be set on the machine launching 
 the pig jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2301) OOM on repair with many inconsistent ranges